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1.  Abstract  /  Executive  Summary 

Fatigue,  stress,  cognitive  overload,  and  other  factors  cause  errors  as  human  operators 
perform  task  operations.  The  goal  of  this  project  was  to  develop,  implement,  and  test  a 
comprehensive  system  for  measuring  and  analyzing  human  performance  related  data. 
Applications  include  basic  psychophysiological  research,  evaluation  of  computer 
interfaces,  evaluation  of  other  task  processes,  and  real-time  performance  monitoring.  The 
developed  system  combines  measurement  and  analysis  of  psychological,  physiological, 
and  performance  measures  into  a  single  system.  The  combination  and  correlation  of  these 
three  factors  provides  for  a  more  robust  and  accurate  assessment  of  total  human 
performance.  The  development  effort  built  off  a  strong  foundation  of  prior  research  and 
development  efforts  for  networked  data  collection,  physiological  monitoring,  eye 
tracking,  operator  workload  modeling,  and  advanced  human-computer  interfaces.  The 
developed  software  system  allows  for  synchronized  collection  of  data  from  any  number 
of  networked  devices  and  provides  an  array  of  signal  analysis  and  display  tools  to  support 
total  performance  assessment.  The  Phase  11  effort  developed  and  implemented  the  human 
performance  based  measurement  system,  and  validated  operation  through  a  series  of 
performance  assessment  experiments.  The  resulting  system  offers  a  comprehensive  set 
tools  for  a  wide  range  of  human  performance  studies  and  related  applications. 
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2.  Identification  and  Significance  of  Research  And  Development 

Fatigue,  stress,  cognitive  overload,  and  other  factors  can  cause  errors  as  hiraian  operators 
perform  various  task  operations.  It  is  therefore  desirable  to  know  when  these  conditions 
occur,  and  what  result  they  are  likely  to  have  on  performance.  For  many  applications,  it  is 
also  desirable  to  be  able  to  identify  individuals  who  can  perform  most  effectively  in 
critical  decision-making  operations,  even  when  confronted  with  an  abundance  of 
information. 


The  overall  goal  of  this  project  was  therefore  to  develop,  implement,  and  test  a 
methodology  and  comprehensive  system  for  measuring  and  analyzing  performance 
related  data  for  human  operator  tasks. 


Initially,  we  envisioned  that  this  technology  would  be  applied  to  the  task  of  systematically 
designing  and  evaluating  interfaces  for  advanced  military  computer-based  operations.  As 
military  computer  systems  and  their  corresponding  interfaces  grow  more  complex,  the 
operator’s  (soldier’s)  task  is  also  becoming  more  difficult.  Interface  designs  that  take  into 
account  the  operator’s  cognitive  processing  can  provide  a  higher  level  of  cognitive 
congruence  between  the  system  the  operator  is  using  and  the  operator’s  existing  internal 
mental  models  of  environment  and  situation. 


As  we  proceeded  through  the  initial  research  stages  of  the  effort,  it  became  clear  that  the 
proposed  system  would  have  significant  benefit  for  virtually  any  human  task 
environment.  We  therefore  broadened  the  scope  of  the  development  effort  to  encompass  a 
the  complete  range  of  performance  assessments,  including  both  physical  and  mental 
factors. 


Targeted  applications  include  basic  research  studies  that  perform  fundamental 
investigations  into  the  nature  of  human  performance.  Or,  a  combination  of  research  and 
development,  such  as  the  evaluation  of  computer  interfaces  or  task  procedures  to  aid  in 
the  design  of  improved  task  procedures.  The  developed  system  could  also  be  used  for 
real-time  performance  monitoring  and  feedback  in  industrial  process  or  other  human  task 
environments. 


The  fundamental  innovation  of  the  developed  system  for  human  operator  assessment 
system  is  the  combined  measurement  and  analysis  of  psychological,  physiological,  and 
performance  measures  in  a  single,  comprehensive,  flexible  system.  The  combination  and 
correlation  of  these  three  factors  within  in  single,  integrated  system  can  provide  for  more 
robust  and  accurate  assessment  of  human  performance.  Figure  1  illustrates  the  concept  of 
fusing  these  three  data  types  to  achieve  a  comprehensive  assessment  of  total  human 
performance. 
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Figure  1:  Total  Human  Performance  Assessment  through  the  Fusion  of  Comprehensive 
Performance-related  Measures 

Physiological  parameters  are  fairly  easy  to  measure,  and  provide  valuable  information 
about  how  a  person  is  reacting  to  the  physical,  mental,  and  other  demands  of  a  given  task. 
These  measures  can  provide  direct  indicators  of  performance  or  may  be  used  to  help 
derive  other  indicators  related  to  performance.  For  instance,  physiological  measures  can 
be  used  to  derive  indicators  of  stress,  fatigue,  attention/alertness,  and  cognitive  workload. 

Psychological  parameters,  while  not  as  directly  measured,  provide  valuable  insight  into 
an  operator’s  cognitive  processes  and  perceptions  and  may  allow  us  to  determine  what 
information  an  operator  is  considering  as  decisions  are  made.  Subjective  feedback  from 
the  human  operator  is  sometimes  the  best  means  for  acquiring  such  information.  Some 
psychological  measures  can  also  be  derived  from  physiological  measures. 

Finally,  task  performance  parameters  are  measures  of  how  an  operator  performs  a  specific 
task  and  how  well  the  task  is  performed.  These  measures  allow  us  to  determine 
methodological  factors  of  performance,  and  to  quantitatively  evaluate  the  effect  that  task 
demand  has  on  an  operator’s  performance.  A  complete  assessment  of  human 
performance  must  incorporate  measurements  from  all  three  of  these  categories. 

Our  development  of  a  highly  flexible  and  comprehensive  system  for  human  performance 
assessment  has  built  off  the  foundation  of  research  and  development  efforts  previously 
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accomplished  at  Cybernet  for  networked  data  collection  systems  and  software, 
physiological  monitoring  systems,  eye  tracking  systems, -operator  workload  modeling,  and 
humam-computer  interface  systems.  The  complete  system  incorporates  measurement  of 
comprehensive  performance  measures  (physiological,  psychological,  and  performance) 
within  a  powerfol,  networked  data  collection  and  analysis  environment.  The  data 
collection  and  analysis  environment  (DCAE)  software  allows  for  synchronized  collection 
of  data  from  any  number  of  networked  devices,  and  provides  an  array  of  signal  analysis 
and  display  tools.  This  concept  is  illustrated  in  Figure  2. 


Figure  2:  High  Level  Representation  of  the  Human  Performance  Measurement  System  - 
integrating  multiple  data  measurement  systems  into  a  comprehensive,  distributed,  data 

collection  and  analysis  architecture 


Primary  measurement  system  components  of  the  developed  system  include  the  portable 
physiological  measurement  system  (PPMS),  the  eye  tracking  system  (ETS),  the  body 
tracking  system  (FireFly)  interface,  and  the  video/audio  capture  mechanism,  which  are  all 
integrated  within  the  Data  Collection  and  Analysis  Environment  (DCAE)  software 
system.  However,  the  flexibility  and  open  architecture  of  this  system  allows  for  easy 
incorporation  of  additional  data  collection  devices/measures  as  well  as  for  integration  of 
customized  data  analysis  and  evaluation  functions. 

The  Portable  Physiological  Measurement  System  (PPMS)  is  a  miniature  computing 
platform  that  performs  multi-channel  data  collection  and  provides  network 
interoperability.  Programmable  signal  conditioning  features  of  the  PPMS  allow  for 
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measurement  of  virtually  any  signal.  The  system  has  a  PCMCIA  interface  providing 
convenient  hardware  configuration  using  PC-cards  for  data  storage  (hard  drives,  FLASH 
cards),  networking  (Ethernet,  wireless  LAN),  and  other  functions.  Currently  two  versions 
of  the  physiological  measurement  system  are  available:  a  16-channel  device  (shown  in 
Figure  3),  and  an  8-channel  device. 


The  eye  tracking  system  provides  for  measurement  of  a  user’s  point  of  regard  (gaze 
location)  and  other  eye-related  parameters,  such  as  pupil  size  and  blink.  The  eye  tracking 
hardware  consists  of  a  head  mounted  eye  tracking  device  (Figure  4)  that  captures  a  video 
image  of  the  eye  as  well  as  a  video  image  of  the  user’s  field  of  view.  The  specialized 
software  system  utilizes  a  client/server  approach,  where  the  user  interface  application 
runs  on  a  client  PC,  and  the  eye  tracking  algorithms  run  on  a  server  PC.  This  two  system 
approach  maximizes  performance  of  the  eye  tracker.  The  eye  tracking  system  may  be 
used  independently  or  as  part  of  the  HPBMS,  where  eye  tracking  results  are  recorded 
through  the  DCAE  software. 


The  Firefly  optical  motion  capture  system  is  designed  to  track  the  motion  of  any  object  (a 
human  body,  vehicles,  models,  etc...),  using  small  infira-red  emitting  tags  placed  on  the 
object.  Figure  5  illustrates  the  basic  concept.  It  is  ideally  suited  for  the  development  of 
computer  games,  the  production  of  computer  animation,  as  a  input  device  for  virtual 
reality  applications,  and  in  providing  data  for  human  biomechanic  or  ergonomic  studies. 
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The  Firefly  may  be  vised  independently  or  as  part  of  the  HPBMS,  where  position  data  is 


Figure  5:  Illustration  of  the  Firefly  System  Operation 


Video  and  audio  signal  capture  in  the  HPBMS  is  performed  through  the  use  of  a  PC- 
controlled  VCR.  The  VCR  can  be  connected  to  the  serial  port  of  any  node  within  the  data 
collection  network.  Initiation  of  video  recording  and  control  of  synchronized  playback  is 
all  performed  through  the  DCAE  software. 

The  DCAE  software  coordinates  all  data  collection  and  analysis  operations  within  the 
HPBMS.  It  allows  for  collection  of  data  fi-om  any  of  the  standard  acquisition  systems 
mentioned  above,  from  acquisition  systems  that  supply  data  through  standard  serial  or 
TCP/IP  protocols,  or  from  custom  developed  acquisition  systems.  The  system  is  operable 
throughout  a  distributed  network  of  collection  nodes,  where  all  data  is  time  and  source 
stamped.  Viewing  and  analysis  tools  allow  for  real-time  viewing  and  processing  of  data 
being  collected  on  any  node  within  the  network.  After  collection,  synchronized  playback 
and  processing  of  all  data  collected  may  then  be  performed.  Further  details  of  this 
comprehensive  software  system  are  presented  throughout  this  report. 

One  particularly  unique  aspect  of  the  developed  human  performance  based  measurement 
system  is  the  ability  to  measure  and  analyze  brain  activity  through  the 
electroencephalogram  (EEC)  signal.  The  EEC  is  a  complex  signal,  representing  the 
summation  of  all  electrical  activity  of  the  brain  measurable  at  the  given  electrode 
location.  Extensive  research  has  been  performed,  and  continues,  in  the  field  of  EEG 
signal  processing,  analysis,  and  interpretation.  There  have  been  many  reports  of 
significant  changes  in  the  EEG  due  to  varying  cognitive  demand/workload.  In  particular, 
a  significant  increase  in  theta  wave  activity  has  been  demonstrated  to  occur  with 
increased  mental  workload  (Wilson  and  Hankins,  1994).  Studies  have  also  demonstrated 
a  suppression  in  alpha  wave  activity  under  certain  types  of  cognitive  tasks.  With  different 
levels  of  sensitivity  in  different  areas  of  the  brain  (Sterman  et  al.,  1993).  From  these  and 
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other  studies,  it  has  also  been  suggested  that  such  measures  can  provide  a  means  for 
evaluating  the  cognitive  capacities  of  different  individuals  (Veigel  and  Sterman,  1993). 
Design  and  implementation  of  the  HPBMS  has  therefore  had  particular  focus  on  the 
development  features  related  to  the  measurement,  analysis,  and  display  of  EEG  signals. 
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3.  Physiological  Assessment  and  Human  Performance 
Measurement  Taxonomy 

3.1  Introduction 

Measurement  parameters  to  be  made  by  the  Human  Performance  Based  Measurement 
System  (HPBMS)  consist  of  physiological  parameters,  psychological  parameters,  and 
performance  parameters.  By  combining  these  measurements  within  a  highly  flexible  and 
integrated  system  we  achieve  a  more  powerful  and  robust  means  of  performing 
performance  assessment. 


Figure  6:  Core  Set  of  Physiological  Measurements 


Measurement 

Parameter 

Measurement 

Technique 

Sensor 

Brain  Activity 

Electroencephalogram 

(EEG) 

Surface  electrodes  on  scalp 
(10-20  system) 

Heart  Rate 

Pulse  sensor;  or  computed 
from  electrocardiogram 
(ECG) 

Finger/ear  clip  (pulse  sensor) 
or  surface  electrodes  on 
chest  (ECG) 

Muscle  Tension 

Electromyogram  (EMG) 

Surface  electrodes  over 
selected  muscle  (i.e. 
trapezius) 

Respiration  Rate 

Chest/abdomen  belt;  or 
impedance  pneumography 

Strain  gauge  or  piezo-electric 
transducer  (belt);  or  surface 
electrodes  on  chest 
(impedance) 

Eye  Motion 

Electrooculogram 

Surface  electrodes  placed 
adjacent  to  eye  socket 

Skin  Impedance 

Surface  electrodes  on  finger 

Event-Related  or 
Evoked 

(stimulated)  Brain 
Response 

Event-Related  Potentials 
(ERPs)  and  Evoked 
Potentials  (EPs)  are 
derived  from  the  EEG 

Surface  electrodes  on  scalp 
(10-20  system).  EPs 
additionally  require 
stimulation  method  (visual, 
auditory,  or  somatosensory) 

Cardiac  Activity 

Electrocardiogram  (ECG) 

Blood  Pressure 

Auto-inflating  pressure  cuff 
on  arm 

BSiiSSBBSMI 

Thermistor 

Placed  on  finger 

Physiological  parameters  include  all  physiological-based  measurements  collected  from 
the  user,  including  brain  activity  (EEG),  heart  rate,  muscle  activity,  etc.  Figure  6  presents 
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the  set  of  core  physiological  measurements  that  were  targeted  for  measxirement  with  the 
developed  system.  However,  given  the  programmable  nature  of  the  system,  virtually  any 
physiological  measurement  may  be  acquired.  The  primary  means  for  acquisition  of  these 
parameters  is  through  the  physiological  measurement  systems. 

Psychological  parameters  can  be  derived  from  physiological  measures,  assessed  through 
subjective  self-reporting  evaluation  (see  Section  3.9),  or  evaluated  by  a  trained  observer. 
These  parameters  will  therefore  be  acquired  either  through  the  physiological 
measurement  system  or  through  the  use  of  evaluation  forms.  For  enhanced  and/or 
automated  analysis  the  forms  may  be  entered  through  the  computer  interface. 

Performance  parameters  include  all  measurements  of  the  user’s  interaction  and 
performance  achievement  for  the  given  task.  These  measurements  will  be  dependent 
upon  the  performance  environment,  the  tasks  involved,  and  the  user  interface  specifics. 
User  interaction  includes  all  measurements  of  the  user’s  activation  and  usage  of  the  user 
interface  tools,  such  as  joysticks,  steering  wheels,  keyboards,  etc.,  or  physical  interaction 
in  a  non-computer  interface  environment.  Performance  parameters  may  include  measures 
such  as  time  for  completion  of  a  task,  accuracy,  or  number  of  errors  committed. 
Performance  and  user  interaction  parameters  can  be  assessed  in  real-time  through  active 
software  running  on  the  user  interface  platform,  through  visual  monitoring  of  the 
procedure,  and/or  through  post-collection  review  of  the  experiment,  as  appropriate. 

While  the  measurement  of  performance  parameters  is  fairly  straightforward,  the 
measurement  and  analysis  of  physiological  and  psychological  parameters  can  be  quite 
complex.  The  following  sections  provide  a  detailed  review  of  psycho-physiological 
measures  and  their  relevance  to  performance  assessment. 

3.2  Electroencephalogram  (EEG)  Measurement  and  Analysis 
3.2.1  Introduction 

The  electroencephalogram  (EEG)  provides  the  information  on  brain  activity  that  occurs 
during  mental  and  physiological  fimctioning,  and  has  been  used  in  the  understanding  of 
mental  workload,  cognitive  processing,  vigilance,  alertness  and  performance.  EEG 
patterns  change  as  a  function  of  arousal  and  alertness,  and  this  phenomenon  is  being  used 
as  a  method  of  assessing  and  monitoring  performance.  The  feasibility  and  utility  of 
topographic  EEG  in  a  multi-task  environment  makes  it  possible  to  associate  cognitive 
changes  with  increased  or  decreased  performance. 

From  scalp  recording  of  brain  electrical  activity  it  becomes  possible  to  reliably  locate 
regions  of  brain  activated  by  specific  cognitive  paradigms  or  functional  activities.  EEG 
scalp  recordings  are  most  often  recorded  utilizing  many  electrodes  in  an  arranged  pattern 
or  montage.  A  common  standard  for  describing  these  position  is  the  International  10/20 
System. 
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Figure  7  shows  the  10-20  International  System  of  electrode  placement.  The  term  10-20  is 
used  because  the  electrodes  are  placed  either  10%  or  20%  of  the  total  distance  between  a 
given  pair  of  skull  landmarks.  The  landmarks  on  the  skull  have  a  topographic  relationship 
with  the  10-20  system  electrodes  and  areas  of  the  cerebral  cortex.  The  relationship 
comprises  five  points  that  are  located  on  the  line  that  connects  die  nasion  to  the  inion: 
Frontal  (Fp),  Occiptial  (O),  Parietal  (P),  Frontal  (F),  and  Central  (C ). 


Figure  7:  The  10-20  International  System  of  Electrode  Placement 

The  10/20  system  allows  the  experimenter  to  associate  changes  in  wave  activity  with 
certain  regions  of  the  brain.  Topographic  maps  show  the  spatial  distribution  and 
significance  of  the  changes  in  these  waves.  Simple  linear  regression  analysis  between 
EEG  power  spectrum  data  and  task  parameters  can  reveal  strong  correlations  with  one 
another;  fiirthermore,  these  correlations  can  be  associated  with  different  regions  of  the 
brain.  These  recordings  can  then  be  placed  onto  a  paper  chart  or  more  commonly 
digitized  into  a  computer  for  frequency  analysis. 


Brainstem 


Figure  8:  Topographic  Relationship  Between  10-20  System  Electrodes  and  Areas  of  the 

Cerebral  Cortex 
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Figure  8  shows  those  areas  of  the  cerebral  cortex  with  relation  to  the  10-20  system  of 

electrode  placement. 

•  The  frontal  lobe:  The  frontal  lobe  is  believed  to  be  the  cortex  of  the  brain  and  indeed, 
it  controls  motor  activity  by  regulating  the  contraction  of  red  muscles.  However,  it  is 
interesting  to  mention  that  some  sensory  data  is  also  processed  by  this  lobe.  Thus, 
"cutting"  this  lobe  (lobotomy)  prevents  the  sensation  of  pain  and  also  inhibits  sexual 
behavior. 

•  Ifre  parietal  lobe:  This  lobe  resides  back  from  the  frontal.  The  parietal  lobe  processes 
most  of  the  sensory  data  that  comes  from  the  body. 

•  The  occipital  lobe:  This  lobe  is  located  at  the  lower  back  of  the  cortex.  It  has  one 
main  function:  processing  all  visual  data.  Therefore  it  also  known  as  the  visual  cortex. 

•  The  temporal  lobe:  This  lobe  is  located  at  the  side  of  the  hemisphere.  It  has  important 
role  in  speaking  and  imderstanding  language,  hearing  and  stability. 


3.2.2  EEG  Frequency  Bands  /  Waveforms 

There  are  many  EEG  waveforms  that  are  indicative  of  both  normal  and  abnormal 
psychological  states.  Lindsley  (1960)  developed  a  classification  system  to  differentiate 
these  wave  forms,  as  shown  in  Figure  9. 
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Figure  9:  Description  of  EEG  Waveforms,  Scalp  Locations  and  Physiological  States 


Type  of 
Waves 

Frequency 

range 

Amplitude 

(Volt) 

Percent  of 
Time  Present 

Regional 
or  Diffuse 

Region  of 
Prominence 

Condition 
when  Present 

Normal  or 
Abnormal 

Alpha 

8-12 

5-100 

5-100 

Diffuse 

Occipital  or 
parietal 

Awake,  relaxed 
Eyes  closed 

Normal 

Beta 

2-20 

5-100 

Diffuse 

Precentral  & 
frontal 

Awake,  No 
movement 

Normal 

Gamma 

30-50 

2-10 

5-100 

Diffuse 

Precentral  & 
frontal 

Awake 

Normal, 

Sleep 

deprived 

Delta 

0.5-4 

20-200 

variable 

Diffuse 

Variable 

Asleep 

Normal 

0.5-4 

20-400 

variable 

Both 

Variable 

Awake 

Abnormal 

Theta 

5-7 

5-100 

variable 

Regional 

Frontal  & 
temporal 

Awake, 
affective  or 
stress  stimuli 

Normal 

Abnormal 

Kappa 

8-12 

5-40 

variable 

Regional 

Anterior  & 
temporal 

Awake, 

problem 

solving 

Normal 

Lambda 

Pos.  or  neg. 

spike  or 
sharp  waves 

5-100 

variable 

Regional 

Parieto¬ 

occipital 

Visual  stimulus 
or  eye  opening 

Normal 

K- 

Complex 

Pos.  sharp 
waves  & 
other  slow 

20-50 

variable 

Diffrise 

Vertex 

Awake, 

auditory 

stimulus 

Normal 

Normal 

pos.  or  neg. 

waves 

50-100 

variable 

Diffuse 

Vertex 

Asleep,  various 
stimuli 

Sleep 

spindles 

12-14 

50-100 

variable 

Regional 

Precentral 

Sleep  onset 

Normal 

The  EEG  itself  is  made  up  of  waves  of  different  frequencies  each  relating  to  different 
aspects  of  mental  life.  EEG  is  usually  described  in  terms  of  these  waveforms:  Gamma 
(>30  Hz),  Beta  (13-30  Hz),  Alpha  (8-13  Hz),  Theta  (4-8  Hz)  and  Delta  (<4  Hz). 
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Delta:  Delta  frequencies  are  the  low  frequency  waveform.  These  are  less  than  4  Hertz  and 
occur  during  sleep  and  in  some  abnormal  processes.  The  presence  of  delta  waves  in  the 
awake  state  is  a  sign  of  brain  abnormality. 

Theta:  Theta  frequencies  are  around  7Hz  and  are  believed  to  reflect  activity  in  the 
hippocampus  and  limbic  system  (a  group  of  structures  deep  within  the  brain  that  are 
generally  believed  to  play  a  role  in  emotions).  They  have  been  implicated  in  drowsiness, 
anxiety,  behavioral  activation  and  behavioral  inhibition.  The  hippocampus  anterior 
portion  of  the  limbic  cortex  helps  to  suppress  the  rage  phenomenon.  For  instance,  if  these 
portions  of  the  limbic  system  are  damaged  a  person  or  animal  becomes  far  more 
susceptible  to  bouts  of  rage. 

Theta  rhythm  also  appears  to  function  normally  to  mediate  and/or  motivate  adaptive, 
complex  behaviors  such  as  learning  and  memory.  If  theta  rhythm  is  not  expressed  most 
strongly  under  unusual  emotional  circumstances  (e.g.  stress;  or  in  disease  states;  e.g. 
schizophrenia;  or  drug  abuse)  there  may  be  an  imbalance  of  three  major  transmitter 
systems.  Thus,  in  the  waking  EEG,  theta  activity  is  an  indicator  of  drowsiness  or  disease. 
With  all  this  in  mind,  the  technicians  observations  and  notations  concerning  the  subjects 
behavior  are  important  during  performance  testing. 

Alpha:  The  most  prominent  feature  of  EEG  is  the  posterior  dominant  rhythm.  It  is  the 
feature  of  brain  electrical  activity  that  was  first  described  in  1929  by  Berger,  who  named 
it  alpha.  This  is  the  8-13  Hz  rhythmic  activity  that  occurs  most  strongly  over  the  occipital 
(back  of  the  head),  parietal,  and  frontal  cortex. 

Contrary  to  popular  belief,  alpha  is  a  common  state  for  the  brain  and  occurs  whenever  a 
person  is  alert  (it  is  a  marker  for  alertness  or  sleep),  but  not  actively  processing 
information.  In  other  words  alpha  activity  is  a  dominant  feature  in  EEG  in  the  awake, 
relaxed  adult.  Alpha  has  been  linked  to  creativity  (creative  subjects  show  alpha  activity 
when  listening  to  and  coming  to  a  solution  for  a  creative  problem),  and  mental  work. 

The  alpha  rhythm  of  an  awake  resting  adult  is  best  seen  when  the  person’s  eyes  are 
closed.  Opening  the  eyes  results  in  an  attenuation  of  the  alpha  rhythm.  Reactivity  to  eye 
opening  is  typically  used  as  evidence  that  the  activity  is  indeed  the  alpha  or  posterior 
dominant  rhythm.  Mental  effort  or  focusing  ones  attention  can  also  attenuate  the  alpha 
rhythm.  The  attenuation  is  dominant  in  the  central  and  parietal  sites  of  the  brain. 

Beta:  The  brain  state  that  most  of  the  brain  is  in  when  we  have  our  eyes  open  and  are 
listening  and  thinking  is  called  beta.  Beta  activity,  which  consists  of  frequencies  higher 
than  13Hz  and  up  to  30  Hz,  is  also  dominant  during  states  of  concentration  and  attention 
activity. 

Beta  tracings  are  best  seen  from  the  anterior  regions,  it  is  commonly  present  in  the 
posterior  regions  as  well,  albeit  masked  by  alpha  rhythm. 
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nGamma:  The  highest  frequency  is  called  gamma  and  contains  waves  of  35Hz  and  up.  In 
contemporary  research  there  is  a  great  deal  of  speculation  that  this  wave  might  reflect  the 
mechanism  of  consciousness. 
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3.2.3  Topographic  Representation 

Topographic  display  and  analysis  of  EEG  data  has  experienced  a  rapid  escalation  in 
utilization  over  the  past  several  years  due  to  the  advanced  tools  and  techniques  that  have 
been  developed.  The  rationale  for  topography  is  that  the  traditional  EEG  tracings  contain 
information  which  under  normal  circumstances  is  not  easily  appreciated  by  the  naked  eye. 
There  is  too  much  data  and  it  is  in  a  form  unsuited  for  visual  analysis. 

The  first  step  for  topographic  representation  is  typically  to  return  to  the  time  domain  after 
a  firequency  analysis  is  made,  and  then  make  an  estimate  of  generators  in  one  period  of 
the  sine  wave  representing  one  Fourier  component  (Weinberg,  et  al.,  1989).  In  this 
instance: 

•  The  first  stage  is  to  record  the  distribution  of  the  electrical  fields  over  the  head. 

•  The  second  stage  is  to  do  a  Fourier  analysis  of  the  recorded  data  for  each  position. 
The  Fourier  an^ysis  is  used  to  extract  a  single  frequency  firom  the  EEG  and  plotted  in 
the  time  domain  for  different  locations  over  the  head 

•  The  third  stage  is  to  select  a  particular  frequency,  or  frequency  band,  of  interest,  e.g., 
8Hz  and  do  the  inverse  computation  for  that  frequency. 

•  The  next  stage  is  to  select  a  method  to  achieve  spatial  resolution  of  the  EEG  signals. 
Spatial  resolution  is  the  ability  of  a  method  to  distinguish  separate  signal  sources  that 
are  positioned  close  to  one  another.  This  is  primarily  achieved  by  increasing  signal-to- 
noise,  and  image  resolution  through  interpolation. 

One  method  for  spatial  resolution  is  the  Laplacian  Derivation;  this  method  involves  the 
computation  of  the  2"^  derivative  in  space  of  the  potential  field  at  each  electrode  position. 
It  converts  the  potential  at  an  electrode  to  a  quantity  that  is  proportional  to  the  current  that 
enters  and  exits  the  scalp  at  that  site. 

Deblurring  is  another  approach  for  spatial  resolution  of  EEG  signals,  using  a  realistic 
biophysical  model  of  the  passive  conductive  properties  of  the  subject’s  head,  to  estimate 
the  electrical  potential  distribution.  The  following  steps  are  followed  in  this  method: 

1 .  measure  the  size  and  shape  of  the  head 
1.  measure  local  scalp  and  skull  thickness 
1 .  estimate  local  skull  conductivity 

1 .  use  a  mathematical  model  to  correct  EEG  signals  for  distortion 

Common  mathematical  models  include  the  finite  element  model,  spherical  model  or  the 
boimdary  integral  model  of  the  cortex,  cerebrospinal  fluid,  skull,  and  the  scalp,  to 
estimate  potentials  that  would  be  recorded  on  the  surface  of  the  brain. 
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Topographical  changes  in  the  EEG  data  as  a  function  ofmental  task  difficulty  were 
studied  by  Fisher  and  Wilson  (1995).  Their  objective  was  to  determine  the  parameters  of 
EEG  associated  with  task  difficulty.  19  scalp  electrodes  were  used  to  record  the  EEG 
data,  and  analysis  was  done  in  a  frequency  range  of  up  to  32  Hz.  The  data  was 
transformed  into  spectral  measures  averaged  for  the  entire  task  spanning  a  duration  of  3 
minutes,  viith  the  tasks  being  considered  as  entities.  Mental  workload  was  assumed  to  be 
a  function  of  three  factors: 

1.  energy  expenditure 

1 .  amount  of  the  available  processing  resources  used 
1.  the  build  up  or  consumption  of  resources 

Topological  evaluation  was  done  by  correlating  the  full  cross-spectral  matrix  for  the  set 
of  signals  from  the  19  scalp  electrodes.  Data  was  displayed  as  topological  brain  maps 
(Duffy, 1986).  Results  showed  a  reduction  in  the  parietal  and  occipital  alpha  power 
during  the  visual  scanning  task,  and  an  association  of  theta  activity  to  attention.  Retinal 
involvement  of  oculo-motor  control  was  attributed  to  the  observations  recorded  with  the 
visual  scanning  task. 


3.2.4  EEG  Experiment  Design 

The  most  important  criteria  in  the  collection  of  EEG  data  is  a  well  controlled 
experimental  design,  which  can  minimize  the  effects  of  demographic  (age,  gender, 
handedness)  and  behavioral  variations  on  the  data. 

Two  types  of  experiments  can  be  used,  namely,  those  where  the  subject’s  attentional  state 
is  not  controlled,  and  others  where  it  is.  In  the  second  type  of  experiments,  the  subject’s 
attention  is  controlled  by  instructing  him/her  to  perform  simple  goal-directed  tasks. 
Additional  factors  that  will  be  controlled  in  these  experiments  are  expectancy,  stimulus 
properties,  cognitive  and  response  requirements,  task  difficulty,  learning,  and  subject 
arousal.  Constancy  of  subject  arousal  level  is  monitored  by  behavioral  criteria  such  as 
accuracy  of  response,  and  the  response-reaction  time.  The  effects  of  fatigue  and  practice 
on  EEG  data  are  minimized  by  manipulating  the  task  difficulty  over  time. 

Another  important  issue  in  experiment  design  is  sample  size.  Due  to  lack  of  knowledge 
on  the  nature  of  data,  a  statistical  power  analysis  cannot  be  performed  easily,  and  instead, 
simple  rules  have  to  be  followed  in  determining  sample  size.  The  number  of  persons  to 
record  data  from,  is  usually  chosen  by  practical  considerations,  with  a  minimum  of  seven 
subjects,  meeting  the  minimum  requirements  of  the  simplest  statistical  tests. 

With  regard  to  the  number  of  data  samples  to  be  collected  per  subject,  common  sense 
“rules  of  thumb”  are  followed.  Sufficient  data  is  gathered  so  that  there  are  at  least  10  data 
observations  for  each  independent  EEG  or  ERP  variable  under  each  category  of  the 
experiment. 
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The  third  factor  in  experiment  design  is  the  length  of  the  EEG  sample,  or  the  number  of 
trials  to  include  in  each  average  ERP.  In  the  case  of  EEG  signals,  it  has  been 
demonstrated  that  30  second  to  2  minute  samples  are  sufficient  to  characterize  the  intra¬ 
subject  variability  of  the  background  EEG.  With  respect  to  ERP  signals,  the  number  of 
trials  must  be  sufficient  to  distinguish  the  Event  Related  Potential  from  the  unrelated 
background  EEG. 

3.2.5  Data  Collection 

The  first  step  in  the  recording  of  EEG  data  is  to  determine  the  optimal  number  of 
electrodes  to  use  and  their  position  on  the  scalp.  With  respect  to  the  number  of  electrodes 
used  for  the  collection  of  EEG  signals,  any  number  from  1  to  256  can  be  used.  The 
greater  the  number  of  electrodes,  the  greater  is  the  spatial  accuracy  and  source 
localization  of  the  signals.  Generally  21  (19  +  2)  electrodes  are  used  in  the  International 
10-20  format.  Ideally  it  is  preferred  that  64  to  512  electrodes  be  used,  with  256  being  the 
optimal  number. 

The  locations  for  placement  of  the  scalp  electrodes  has  been  standardized  since  1958 
(Jasper,  1958)  through  a  system  called  the  International  10-20  system.  This  system  uses 
four  reference  points:  the  inion  (small  bump  at  the  back  of  the  head),  the  nasion  (small 
cavity  at  the  base  of  the  nose),  and  the  right  and  left  preauricular  (mastoid)  points  (tiny 
cavities  above  and  behind  each  ear),  and  19  active  sites.  For  standard  EEGs,  the  linked 
mastoid  references  are  commonly  used.  The  organization  of  electrode  placement  on  the 
scalp  is  illustrated  in  Figure  10. 
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Figure  10:  Electrode  Locations  in  the  International  10-20  System 


Electrodes  are  placed  on  the  scalp  at  points  that  are  10  and  20  percent  of  the  distance 
from  the  lines  of  the  nasion  to  the  inion,  and  from  the  left  to  the  right  mastoid  points.  The 
capital  letters  F,  T,  P,  O  and  C  marking  each  electrode  location  refer  to  the  frontal, 
temporal,  parietal,  and  occipital  lobes  and  the  central  sulcus  respectively.  Odd  numbered 
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channels  represent  locations  on  the  left  and  even  numbers  the  locations  on  the  right  side 
of  the  brain.  The  full  10-20  montage  involves  all  19  electrodes  additionally 
supplemented  with  two  leads  to  record  eye  movements  (EOG),  making  it  a  total  of  21 
channels  to  record.  Eye  movements  are  recorded  since  they  produce  artifacts  in  the  EEG 
recordings.  Other  artifacts  are  caused  by  eye  blinks,  head  movements  and  muscle  activity 
especially  of  the  temporalis  and  the  masseter. 

An  important  challenge  in  the  use  of  multiple  electrodes  is  to  determine  the  accurate 
spatial  positions  of  these  electrodes.  Determining  the  position  of  the  electrodes  on  the 
scalp  helps  in  the  correlation  of  the  EEG  signals  with  the  underlying  brain  anatomy.  One 
method  developed  by  Gevins  (1992)  uses  a  flexible  grid  of  electrical  field  sources  or 
nodes  in  close  proximity  to  the  head,  and  measures  the  potential  induced  in  each  electrode 
when  each  node  in  the  grid  is  activated.  He  suggested  the  use  of  resonant  circuits  or 
switch  points  as  nodes,  which  formed  a  grid  on  a  flexible  cap.  The  relative  position  of 
each  node  is  calculated  by  interpolation  fi-om  the  pattern  of  induced  voltages. 

EEG  data  can  be  recorded  in  two  ways  namely  as  analog  data,  or  as  directly  digitized 
data.  The  advantages  of  direct  digitization  of  EEG  signals  over  analog  recording  are 
several,  the  most  important  of  which  is  the  avoidance  of  noise  patterns  that  resemble 
brain  waves  (e.g.  theta  band  activity). 

Signal  conditioning  is  an  essential  process  in  the  recording  of  EEG  data.  Linear-phase 
filters  with  high  attenuation  outside  their  passband  (for  example,  24  dB  or  more 
attenuation  per  octave  outside  their  passband)  are  used.  There  are  several  cutoff  values 
for  EEG  signal  conditioning,  the  most  common  bandpass  being  0.5  to  70  Hz  for  EEG 
studies.  The  bandpass  values  of  the  ERP  signals  are  as  follows: 

1.  Brain  Stem  ERP:  50-  150  Hz  to  3000  Hz 

2.  Exogenous  ERPs  including  pattern  shift  visual  ERP:  1  to  100  -  300  Hz 

3.  Endogenous  ERP  studies:  DC  0.01  to  16  Hz 

Sampling  rate  for  digitization  needs  to  be  at  least  tvvice  the  highest  frequency  at  which  the 
filtered  potentials  have  negligible  power.  This  frequency  is  also  called  iheNyquist 
frequency. 

The  conversion  of  EEG  signals  to  a  digital  representation  is  usually  done  with  12  bits  of 
accuracy,  between  plus  and  minus  2048,  or  4096  levels  of  quantization.  In  order  to 
represent  the  EEG  with  maximal  accuracy,  the  amplitude  of  the  largest  waveform  that 
needs  to  be  recorded  without  distortion  should  span  most  of  the  numerical  range  of  the 
A/D  converter.  It  is  also  essential  not  to  exceed  this  numerical  range  in  order  to  avoid 
saturation  of  the  A/D  converter’s  range.  It  is  advisable  to  save  some  spare  A/D  range  to 
allow  for  changes  in  the  DC  level  of  the  signal. 

For  example  if  abnormal  adult  delta  band  EEG  activity  has  to  be  recorded,  the  full  A/D 
range  should  correspond  to  plus  or  minus  500  pV  (600  pV  for  the  EEG  and  400  pV  for 
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the  spare  A/D  range).  If  the  full  scale  input  range  of  the  converter  is  plus  or  minus  5  V, 
each  microvolt  of  EEG  will  correspond  to  0.01  V  at  the  A/D  converter.  Each  microvolt 
will  be  represented  by  4  quantization  levels  when  a  12  bit  digitization  is  used. 
Optimization  of  the  range  of  the  A/D  converter  can  be  achieved  by  adjusting  the 
sensitivity  of  the  EEG  amplifiers  with  a  well  chosen  test  signal,  and  monitoring  the 
voltage  at  the  input  to  the  A/D  converter. 

Another  factor  that  requires  consideration  is  the  finite  amount  of  time  required  to  digitize 
the  signal,  resulting  in  a  time  difference  between  sampling  the  first  and  last  channel.  An 
imwanted  difference  in  sampling  time  can  systematically  distort  the  precise  measurements 
of  interchannel  phase  and  time  delay.  This  can  be  remedied  in  two  ways,  namely,  using  a 
“sample  and  hold”  A/D  converter  wWch  stores  the  voltage  of  all  the  channels  at  the  same 
instant,  or  following  the  time  consuming  procedure  of  numerically  interpolating  the 
digitized  values  to  compensate  for  the  systematic  time  shift  during  the  digitization. 

3.2.6  Data  Preparation 

There  are  several  steps  involved  in  data  preparation  prior  to  analysis  of  the  digitized  EEG 
signals.  These  include: 

1.  Topographic  representation 

2.  Digital  filtering 

3.  Numerical  calibration 

4.  Artifact  rejection 

5.  Creation  of  data  sets. 

Topographic  representation 

Topographic  representation  was  previously  discussed  in  Section  3.2.3. 

Digital  Filtering 

Digital  filtering  is  a  process  by  which  imwanted  frequency  components  are  removed,  or 
frequency  bands  of  interest  are  isolated  fi’om  a  signal.  For  example,  contamination  of  the 
EEG  signal  by  the  50  Hz  or  60  Hz  power  lines,  as  well  as  their  harmonic  frequencies  can 
be  removed.  There  are  simple  software  programs  that  apply  a  series  of  weighting  and 
summing  arithmetic  operations  according  to  precise  formulas  (Oppenheim  and  Schafer 
1975;  Rabiner  and  Gold  1975;  Peled  and  Liu  1976;  Oppenheim  et  al.  1983).  Programs 
for  designing  digital  filters  with  user-specified  bandpass  and  stopband  parameters  are 
available  from  IEEE  (1979).  In  general,  non-phase-shifting  filters  (non-recursive  filters) 
are  used  in  order  to  measure  the  timing  relationships  between  channels. 

Numerical  Calibration 

Niunerical  calibration  is  a  procedure  to  determine  the  gain  correction  factor  that  has  to  be 
applied  to  each  sample  of  the  digitized  data.  This  is  done  by  passing  a  calibration  signal 
through  the  amplifiers,  filters,  and  the  A/D  converter.  Single  frequency  sine  waves, 
square  waves,  and  swept-frequency  sine  waves  are  commonly  used  as  calibration  signals. 
The  choice  of  the  signal  depends  on  the  purpose  of  the  analysis.  Sine  waves  resemble 
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ongoing  brain  wave  activity  and  can  pass  through  a  recording  system  without  distortion. 
Square  waves  contain  abrupt  transitions  that  mimic  the  effects  of  paroxysmal  events, 
while  swept  frequency  waves  provide  gain  calibration  at  the  high  and  low  cutoff 
frequencies  in  passband  amplifiers  and  filters.  It  is  standard  practice  to  record  a 
calibration  signal  at  the  beginning  of  every  recording  session,  and  to  apply  the  correction 
factor  to  each  channel  of  data  to  compensate  for  inter-channel  differences  in  gain. 

When  a  new  system  is  first  installed,  it  is  good  practice  to  perform  a  sweep  of  the  entire 
frequency  range  of  the  recording  system’s  passband  to  determine  if  different  gain  factors 
need  to  be  applied  for  the  different  frequencies.  Very  long  recordings  should  also  be 
made  to  determine  the  magnitude  of  temporal  variations.  If  it  is  found  that  the  signal  is 
significantly  attenuated  or  amplifies  at  the  edges  of  the  desired  passband,  it  is  often 
simpler  to  record  with  a  broader  bandwidth,  and  digitize  at  a  higher  rate,  than  to  perform 
frequency  specific  calibration  corrections. 

A  second  type  of  calibration  correction  is  performed  for  significant  phase  differences 
between  channels,  a  procedure  that  is  advisable  when  studying  inter-channel  time  delays. 
These  differences  are  measured  by  sending  the  same  signal  through  all  the  amplifiers,  and 
computing  the  correction  coefficients  based  on  any  cross-spectral  phase  differences. 

Another  type  of  calibration  signal  is  a  zero  level  input  obtained  by  shorting  all  the  input 
channels  to  ground,  prior  to  the  A/D  converter,  which  allows  the  measurement  of  noise 
and  the  DC  offset  of  the  amplification  and  recording  system  due  to  thermal  noise  in  the 
amplifiers,  loose  or  dirty  electrical  connections,  and  ambient  electrical  or  magnetic  fields. 

Artifact  Rejection 

Removal  of  artifacts  from  EEG  data  is  an  essential  step  prior  to  data  analysis. 

Traditionally  visual  inspection  of  data  has  been  used  for  artifact  removal,  a  procedure  that 
is  prone  to  subjective  bias.  There  are  several  sources  of  artifacts  in  the  recorded  EEG 
signals.  Drowsiness  of  the  subject  during  the  experimental  session  can  cause  an  artifact 
that  can  be  confused  with  a  pathological  slowing  of  the  EEG.  The  three  major  classes  of 
artifacts  include: 

1 .  Head  and  body  movements,  perspiration,  and  low  frequency  instrument  artifacts 
(under  1  Hz), 

2.  Eye  movements  (under  3  Hz  in  the  frontal  channels),  and 

3 .  High  frequency  artifacts  including  gross  muscle  potentials  from  scalp  and  tongue 
muscles  (34  -  50  Hz). 

In  most  instances  the  best  method  for  removing  artifacts  is  by  visual  inspection.  However 
for  the  piupose  of  removing  eye  movement  potentials  (EGG),  subtraction  of  a  weighted 
and  simultaneously  recorded  EOG  signal  from  each  channel  of  EEG,  is  currently 
followed.  One  of  the  more  popular  artifact  rejection  algorithms  is  short-time  spectral 
subtraction  developed  by  Whitton  et  al.  (1978).  In  this  method,  the  differences  in  the 


20 


CYBERNET  SYSTEMS  CORPORATION 


spectral  signatures  of  eye  movement  and  brain  signals  are  used  to  create  filters,  which 
then  remove  eye  movement  contaminants  from  each  trial. 

A  recent  adaptive  interference  canceler  for  EEG  movement  and  eye  artifact  has  been 
developed  by  Gevins  et  al.  (1996).  A  sketch  of  the  system  is  illustrated  in  Figure  11.  The 
head  and  body  movement  reference  signal  is  provided  by  an  accelerometer  or  motion 
detector,  and  the  spatial  averages  of  the  EEG  channels.  Eye  motion  sensors  are  used  as  a 
reference  for  eye  movement  artifacts.  A  composite  reference  signal  consists  of  the  head, 
body,  and  eye  movement  reference  signals.  The  contaminated  EEG  is  the  primary  input 
to  the  adaptive  movement  and  eye  artifact  canceler,  in  which  an  adaptive  filter  estimates 
the  contaminants  in  the  measured  EEG  data,  and  subtracts  them  from  the  primary  signal 
to  obtain  the  corrected  EEG  data.  This  methodlogy  is  discussed  in  detail  in  Appendix  B. 


Primaiy 


Figure  11:  Adaptive  Interference  Canceler 
(Gevins  et  al,  1996) 


Data  Set  Formation 

The  final  step  in  data  preparation  is  to  look  for  outliers,  and  to  check  that  the  data  meet 
the  a  priori  specifications  for  experiment  control.  This  process  involves  observation  of 
behavioral  measures  of  the  subject,  such  as  reaction  time  or  response  accuracy,  for  each 
observation  or  trial.  If  the  behavioral  data  is  significantly  out-of-range,  the  EEG  and  ERP 
data  collected  during  this  segment  can  be  discarded. 
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3.2.7  Primary  Analysis 

Following  the  preliminary  data  preparation  steps  of  signal  conditioning,  digitization, 
artifact  rejection,  and  formation  of  data  sets,  primary  analysis  is  done  to  extract 
meaningful  information  from  the  data.  The  major  objects  of  primary  analysis  include  the 
analysis  of  backgroimd  EEG,  event-related  signals,  and/or  spatial  processes. 

Background  EEG  can  be  analyzed  in  two  ways  namely,  frequency  analysis  and  non¬ 
stationary  analysis.  The  most  common  method  of  characterizing  the  frequency  content  of 
background  EEG  is  by  the  use  of  Fourier  analysis,  which  results  in  the  power  and  phase 
spectra  of  the  EEG  data.  The  power  spectrum  results  from  taking  the  sum  of  the  energy 
of  the  sine  and  cosine  waves  at  each  frequency,  while  the  phase  spectrum  is  the  inverse 
tangent  of  their  ratio . 

The  results  of  spectral  analysis  are  often  grouped  into  traditional  frequency  bands, 
namely,  delta  (less  than  4  Hz),  alpha  (8  - 13  Hz),  beta  (14  -  30  Hz),  and  gamma  activity 
(above  30  Hz).  However,  the  use  of  these  traditional  frequency  bands  has  to  be  based  on 
the  types  of  experiments  that  are  conducted.  For  example,  in  the  case  of  EEG  recorded 
from  children,  the  delta  and  theta  bands  may  have  to  be  divided  into  additional  bands. 
Also  other  measures  derived  from  the  spectrum,  such  as  the  peak  frequency  in  each  band 
may  be  more  sensitive  to  specific  experimental  manipulations  than  the  energy  in  a  band. 

The  most  popular  method  of  performing  frequency  analysis  is  the  application  of  the  Fast 
Fourier  Transform  (FFT)  algorithm  directly  to  short  segments  of  digitized  data,  usually  1 
to  4  seconds  long.  These  digitized  time  series  segments  have  to  be  prepared  for  the  FFT 
by  first  removing  any  non  zero  mean  and  trend  in  the  segment,  and  then  smoothing  the 
abrupt  discontinuity  at  the  edges  of  the  segment  to  avoid  spurious  spectral  components. 

A  detailed  description  of  the  method  of  performing  spectral  analysis  is  provided  in  the 
following  section. 

From  a  mathematical/statistical  point  of  view,  the  EEG  represents  a  set  of  continuous 
voltage/time  graphs  and  can  therefore  be  considered  as  a  multivariate  time  series.  Like 
many  other  biophysical  phenomena,  the  EEG  shows  more  or  less  irregular  patterns  and, 
therefore  can  only  be  described  by  average  properties.  Since  EEG  belongs  to  a  category 
of  random  data  that  cannot  be  described  in  explicit  mathematical  relations,  it  is  described 
in  statistical  terms. 

The  analytical  approach  is  to  define  a  random  process  x(t)  generating  “realizations”, 
which  may  be  thought  of  as  random  functions,  having  statistical  features  as  close  as 
possible  to  those  of  the  observed  data.  A  random  (or  stochastic)  process  represents  the 
collection  of  all  possible  sample  functions  which  the  random  phenomenon  might  have 
produced.  Such  a  collection  is  called  an  ensemble.  A  sample  record  of  EEG  data  may  be 
regarded  as  a  single  realization  of  a  random  process,  and  thus  statistical  testing  of  this 
process  becomes  possible. 
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Appendix  C  addresses,  in  greater  detail,  the  methodologies  involved  in  the  primary 
analysis  of  EEG. 

3.2.8  Feature  Extraction  and  Data  Standardization 

Following  Fourier  transformation  of  the  EEG  data  (or  other  primary  analysis  technique), 
further  condensation,  parameter  extraction,  synopsis  and  statistical  treatment  of  the  EEG 
data  are  necessary.  There  are  several  techniques  to  perform  the  post  processing  of 
spectral  data,  and  these  are  listed  as  follows: 

Parameter  Extraction 

•  Band  power,  band  coherence 

•  Power  of  white,  pink  and  colored  noise  components 

•  Baseline  slope  and  intercept 

•  Spectral  quotients 

.  Peak  parameters:  peak  frequency,  peak  power  (area  of  the  peak),  relative  spectral 
peak  power  (RSPP) 

•  half  power  bandwidth,  peak  width  (distance  between  the  peak  footpoints), 

•  skewness  (asymmetry  coefficient),  kurtosis  (peak  shape) 

Statistical  Treatment  / Pattern  Recognition 

•  Simple  treatment 

•  Mean  and  standard  deviation  of  spectral  parameters 

•  Average  spectra  with  range  of  variance 

•  Product  moment  correlation  of  power  spectra 

•  Analysis  of  variance 

•  Multivariate  analysis 

•  Principle  component  and  factor  analysis 

•  Multidimensional  scaling 

•  Multivariate  analysis  of  variance 

•  Discriminant  analysis 

•  Cluster  analysis 

•  Non-linear  pattern  classification 

Condensation  and  Visual  Presentation 

•  Sequential  spectral  or  coherence  displays 

•  Compressed  spectral  arrays 

•  Contour  plots 

•  Topographic  parameter  display 

•  3-Dimensional  scalp  maps 

•  Chemoff  faces 

Methodologies  for  parameter  extraction  are  discussed  in  detail  in  Appendix  D,  while 
statistical  pattern  recognition  techniques  are  discussed  in  Appendix  E. 
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Key  Issues  in  using  Pattern  Recognition  (PR)  Algorithms 

Several  important  issues  constitute  the  key  to  effective  use  of  PR  algorithms.  These 
include  having  sufficient  sample  of  data,  choosing  good  features  and  good  combinations 
of  feature  subsets,  standardizing  features,  validating  and  assessing  the  significance  of  a 
classification  equation,  and  interpreting  results. 

Sample  size'.  The  sample  size  required  to  achieve  a  given  level  of  significance  in 
distinguishing  between  conditions  generally  has  an  inverse  relationship  with  the  degree  of 
difference  between  conditions.  As  differences  between  conditions  increase,  smaller 
sample  sizes  are  sufficient  to  obtain  a  given  significance  level.  However,  it  is  also 
possible  that  increasing  the  sample  size  will  decrease  the  differences  between  conditions 
because  of  the  individual  differences  in  the  group  of  persons  tested.  This  indicates  that 
the  smaller  sample  is  insufficient  to  characterize  within-condition  variability.  Since  these 
factors  cannot  be  estimated  a  priori,  a  reasonable  procedure  is  to  measure  the 
approximate  magnitude  of  separation  between  conditions  on  several  small  groups  of 
persons.  If  the  separation  varies  greatly  between  the  groups,  a  larger  sample  must  be 
used.  Cohen  (1969)  has  shown  that  with  this  information,  an  estimate  of  the  minimal 
necessary  sample  size  can  be  made  using  a  table  for  statistical  power  analysis. 

Parametric  studies  for  variable  subset  selection:  Feature  selection  is  problematic  since  it 
is  not  clear  which  ones  are  the  best,  or  how  to  combine  them  to  produce  the  highest 
discrimination  between  the  conditions  of  an  experiment.  A  reasonable  procedure  is  to 
perform  a  series  of  parametric  studies  on  a  small  but  representative  data  set.  In  each 
study  a  different  type  of  feature  is  input  to  a  classifier-directed  feature  extraction 
algorithm,  and  the  one  with  the  best  performance  is  selected.  Exhaustive  searches  can 
then  be  performed  to  find  the  best  subset  of  the  best  feature  type. 

Standardization  of  features:  A  simple  but  effective  method  of  amplitude  standardization 
can  be  obtained  by  transforming  the  voltage  of  each  channel  of  each  participant’s  data 
into  a  standard  score;  the  mean  is  subtracted  and  the  result  is  divided  by  the  standard 
deviation  of  all  observations  fi:om  all  conditions  for  the  participant.  A  method  of 
temporal  standardization  for  the  ERPs  can  be  achieved  by  applying  PCA  to  the  averaged 
time  series  pooled  across  channels  and  conditions,  either  for  each  person  separately,  or 
for  all  participants.  Alternatively,  analysis  windows  can  be  centered  on  each  person’s 
ERP  peaks.  PCA  can  also  be  similarly  applied  to  EEG  spectra  to  define  common 
frequency  bands. 

Validation  and  assessing  sisnificance:  Validation  is  an  important  step  in  checking  the 
validity  of  the  equations  produced  by  the  PR  algorithm.  This  check  reduces  the 
possibility  of  a  type  I  (false  positive)  error  due  to  insufficient  data  samples,  or  to 
violations  of  the  statistical  distribution  assumptions  of  the  analyses.  The  most  convincing 
analyses  is  obtained  with  a  completely  independent  data  sample  drawn  fi-om  a  new  set  of 
controls.  However  because  of  the  expense  and  the  difficulty  in  collecting  data,  validation 
is  often  performed  using  a  portion  of  the  database,  half  or  one  third.  This  allows  for 
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testing  of  the  PR  equation  generated  from  the  rest  of  the  data.  This  process  can  be 
performed  by  different  shuffling  of  the  training  and  the  validation  data  sets.  Then  the 
average  classification  accuracy  obtained  on  the  validation  data  sets  can  be  used  for 
assessing  the  significance  of  the  results. 

There  are  several  methods  for  assessing  the  significance  of  the  validation  classification 
accuracy.  These  procedures  can  be  used  alone  or  in  combination.  One  such  method 
involves  determining  the  accuracy  of  the  PR  equation  on  randomized  data,  i.e.  data  whose 
identity  by  experimental  condition  has  intentionally  been  randomly  scrambled.  The  same 
observations  which  were  used  to  develop  the  PR  equation  are  randomly  arranged,  a  new 
PR  equation  is  computed,  and  the  significance  of  a  similarly  randomly  ordered  validation 
data  set  is  then  measured.  This  procedure  is  repeated  a  number  of  times  using  data  from 
different  electrodes,  or  different  temporal  epochs,  resulting  in  an  estimate  of  the  mean  and 
the  variance  of  the  significance  level  of  classification  of  randomized  data.  It  is  then  a 
simple  matter  to  compare  the  results  obtained  vwth  the  correctly  arranged  data  with  those 
obtained  with  the  random  data  to  assess  the  former’s  actual  significance. 

Interpretins  results:  Neurophysiological  interpretation  of  the  PR  equations  and  their 
coefficients  requires  that  the  features  not  be  randomly  combined.  The  use  of 
spatiotemporal  constraints  can  greatly  facilitate  an  anatomical  interpretation  of  the  results, 
^^en  this  is  done,  separate  equations  are  derived  for  each  electrode,  and  possibly  for 
each  time  interval  when  analyzing  ERPs.  Comparison  of  the  relative  significance  of  the 
equations  can  reveal  information  about  the  topography  and  the  timing  of  the  effects  under 
consideration.  When  standardized  features  are  used,  (mean  =  0,  and  variance  =  1), 
equations  with  many  terms  can  be  reduced  by  extracting  the  largest  terms  and  then 
reducing  the  equations  to  the  minimal  number  of  terms  required  to  produce  a  significant 
classification. 

3.2.9  Estimation  of  Event-related  Signals 

There  are  several  ways  of  estimating  an  event-related  signal.  These  include  averaging, 
Wiener  filtering,  latency  correction,  time-varying  filtering,  pre-  to  post-stimulus 
transformation,  and  Wigner  distribution. 

Averaging:  This  is  a  technique  that  is  useful  for  the  analysis  of  event-related  activity.  In 
this  method,  a  number  of  brief  EEG  time  series  which  are  time  registered  to  an  expected 
or  actual  stimulus  or  response  are  averaged.  Some  of  the  inherent  disadvantages  to  the 
averaging  technique  are  that  the  event-related  signal  may  not  occur  at  the  same  time,  may 
not  have  exactly  the  same  shape,  or  may  not  be  present  to  the  exact  degree  in  each  trial. 

Wiener  filtering:  This  method  also  known  as  the  minimum  mean  square  error  (MMSE) 
filtering,  is  a  controversial  approach  to  improving  event-related  signal  estimation 
(McGillem  et  al.,  1981).  It  works  on  the  principle  of  using  a  custom  designed  digital 
filter  which  selectively  enhances  the  event-related  signal  components  and  suppresses  the 
unrelated  noise.  In  order  to  implement  the  MMSE  filters,  it  is  necessary  to  have  an 
estimate  of  the  signal  or  its  spectrum,  and/or  of  the  noise.  This  is  a  problem  since  the 


25 


CYBERNET  SYSTEMS  CORPORATION 


actual  signal  is  unknown.  This  is  overcome  by  first  estimating  the  spectrum  of  the  signal 
plus  noise  followed  by  the  spectrum  of  the  noise  alone.  .The  spectrum  of  the  noise  alone 
is  estimated  by  subtracting  the  power  spectrum  of  the  average  (signal  +  1/N  times  noise) 
fi:om  the  average  of  the  set  of  single-trial  power  spectra  (signal  plus  noise).  The  noise  can 
also  be  estimated  by  averaging  the  residuals  resulting  fi'om  subtraction  of  the  average 
ERP  firom  each  trial,  or  by  averaging  with  the  polarity  reversed  on  alternate  trials.  Wiener 
filtering  is  not  a  commonly  used  method  due  to  mixed  success  of  the  results. 

Latency  correction:  This  is  another  method  for  estimating  event-related  signals  that  can 
occur  at  different  times  on  different  trials.  This  procedure  adjusts  the  temporal  position 
of  each  trial  to  best  align  it  with  a  peak  in  the  average  ERP.  This  is  done  by  shifting  the 
entire  time  series  of  each  trial  or  segments  of  it  by  the  lag  number  of  the  maximum 
correlation  of  the  time  series  with  the  average.  The  maiii  drawback  of  this  procedure  is 
that  it  requires  the  event-related  signal  to  be  large  so  that  individual  trials  do  not  line  up 
with  the  background  EEG  signals.  An  essential  accompaniment  to  the  latency  correction 
procedure  is  reliability  testing  for  the  signal-to-noise  ratio,  as  well  as  for  the  accuracy  of 
the  time-shift  estimates. 

Time-varying  filtering:  This  is  a  technique  to  deal  with  time- varying  ERP.  The  averaged 
ERP  time  series  is  decomposed  by  a  bank  of  octave-band  digital  filters  with  time-varying 
gain  coefficients,  and  the  event-related  signal  is  estimated  by  summing  the  weighted  filter 
outputs.  A  more  recent  approach  by  Yu  and  McGillem  (1983)  is  the  application  of  a 
time-varying  MMSE  filter  to  each  trial.  This  approach  is  based  on  an  explicit  model  of 
the  ERP  as  a  random  process  with  varying  amplitude  and  timing,  and  which  is 
contaminated  by  random  noise. 

Pre-  to  post-stimulus  transformation:  This  model  proposed  by  Basar  (1980)  assumes  that 
the  event-related  signal  is  a  nonlinear  transformation  of  the  brain  state  existing  prior  to 
the  time  of  stimulation.  Several  preliminary  applications  of  this  concept  to  practical 
signal  processing  algorithms  have  resulted  in  encouraging  results  (Rarmer  et  al.  1983). 

Wigner  distribution:  More  recently,  the  Wigner  distribution  has  been  applied  by  Morgan 
and  Gevins  (1986),  which  allows  the  visualization  of  both  the  time  and  the  frequency 
aspects  of  a  set  of  data.  With  this  method,  it  is  possible  to  view  the  spectral  patterns  of 
rapidly  changing  event-related  processes  within  small  frequency  regions.  These  above 
mentioned  methods  are  some  of  the  ways  in  which  event-related  signals  can  be  estimated 
and  analyzed. 

3.2.10  Spatial  Analysis 

Spatial  analysis  is  an  important  aspect  of  EEG  research  since  the  spatial  patterning  of 
scalp-recorded  brain  signals  is  complex,  reflecting  the  activity  of  several  anatomically 
and  functionally  differentiated  underlying  neural  systems.  The  separation  between 
electrodes  required  to  avoid  spatial  aliasing  can  be  determined  by  measuring  the  spatial 
spectrum  from  closely  spaced  electrodes.  The  average  inter-electrode  distance  while 
using  60  electrodes  is  3.25  cm,  while  with  120  electrodes  is  2.3  cm. 
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The  first  step  of  spatial  analysis  has  been  to  examine  the  differences  in  EEG  or  ERP 
voltage  or  the  EEG  spectral  intensity  between  the  various  electrodes.  Over-the-head 
spatial  displays  represent  an  important  development  that  allowed  visualization  of  the 
EEG  and  ERP  potential  distribution  in  two  dimensions.  Maps  of  this  type  consist  of  a 
series  of  contour  lines  representing  the  peaks  and  valleys  of  potential,  very  similar  to  the 
topographic  geophysical  maps.  The  three  major  issues  associated  with  topographic  maps 
include:  1)  Manner  of  projecting  the  three-dimensional  head  onto  a  two-dimensional 
representation,  2)  Choice  of  reference  electrode,  and  3)  Type  and  the  amount  of 
interpolation.  In  general,  the  contours  are  drawn  on  a  sketch  of  a  top,  side,  or  back  view 
of  the  head  using  orthographic  projection  and  interpolating  the  voltage  values  at  several 
points  between  the  actual  recorded  electrodes. 

An  important  factor  that  affects  the  topographic  voltage  distribution  is  the  choice  of  the 
different  reference  electrodes.  The  solution  to  the  reference  problem  of  spatial  maps  is  to 
convert  the  measurements  into  quantities  that  are  independent  of  the  reference,  for 
example,  to  gradients  by  taking  the  first  derivative  of  the  field  (Remond  1968),  to  current 
densities  by  applying  the  Laplacian  operator,  or  to  measures  of  the  extrema  of  field  power 
(Lehmann  1971).  Recently  color  computer  graphics  have  become  very  popular  for 
displaying  the  scalp  voltage  distribution  of  16  or  more  chaimel  ERPs  or  EEG  spectral 
intensities. 

While  valuable  information  can  be  obtained  by  viewing  over-the-head  displays,  statistical 
methods  are  necessary  to  quantitatively  compare  the  data.  There  are  several  approaches 
to  this  problem.  The  maps  can  be  reduced  to  a  few  features,  such  as  the  location, 
orientation  and  magnitude  of  an  equivalent  current  dipole  which  could  produce  the 
observed  field,  or  to  the  location  and  magnitude  of  the  maximal  and  minimal  values  of 
the  field  power.  Another  approach  to  objectifying  the  interpretation  of  the  maps  is  to 
statistically  compare  them.  Multivariate  statistical  analysis  is  used  by  several  systems 
such  as  the  BEAM  system  by  Duffy  et  al.  (1981),  the  Brain  State  Analyzer  by  John  et  al. 
(1977),  and  the  Neurocognitive  Pattern  Analysis  developed  at  the  EEG  Systems 
Laboratory.  The  BEAM  system  works  by  reference  to  normative  values  derived  firom 
control  populations  for  each  electrode,  while  the  Brain  State  Analyzer  has  an  additional 
component  of  comparing  serial  recordings  firom  the  same  patient.  The  Neurocognitive 
Pattern  Analyzer  compares  between  experimental  conditions  for  each  electrode  or  group 
of  electrodes  for  one  or  more  people. 

The  reduction  of  spatial  smearing  of  brain  potentials  due  to  the  effects  of  volume 
conduction  is  highly  desirable,  since  the  hypothetical  point  source  of  current  on  the 
surface  of  the  cortex  becomes  spread  over  an  area  of  approximately  2.5  cm  when 
transmitted  to  the  scalp  (Doyle  and  Gevins  1986).  This  is  achieved  by  the  use  of  a 
Laplacian  operator  montage  that  requires  the  computation  of  the  curvature  of  the  potential 
field  at  each  electrode.  This  amoimts  to  adding  together  bipolar  EEG  channels 
surrounding  a  particular  electrode  after  they  have  been  weighted  by  the  actual  inter¬ 
electrode  distances.  This  transformation  has  an  additional  advantage  of  removing 
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dependence  of  the  measures  on  the  reference  electrodes  by  converting  the  scalp  voltage 
distribution  into  measures  of  the  flow  of  current  entering  and  exiting  from  the  head  at 
each  electrode.  However,  this  Laplacian  montage  (LM)  or  current  source  density  (CSD) 
works  best  when  many  channels  are  recorded  since  it  is  not  possible  to  accurately 
compute  the  LM  for  channels  on  the  periphery.  For  example,  with  19  channels  of  EEG, 
only  9  LM  channels  can  be  computed,  while  for  63  channels,  43  LM  channels  can  be 
computed. 

Another  promising  method  is  the  use  of  mathematical  transformations  such  as  spatial 
deconvolution,  which  can  greatly  reduce  the  spatial  smearing  of  scalp  recorded  potentials, 
making  it  appear  as  if  the  signals  were  recorded  on  the  cortical  surface. 

While  the  observation  and  the  analysis  of  spatial  patterns  is  straight  forward,  the  basic  but 
more  difficult  problem  is  to  determine  the  source  in  the  brain  of  the  patterns  observed  on 
the  scalp.  For  this  purpose,  a  priori  models  of  the  character  and  the  number  of  sources 
have  been  proposed.  The  most  commonly  used  model  assumes  that  the  scalp  field 
distribution  arises  from  an  equivalent  current  dipole  source,  that  can  be  represented  by  six 
parameters,  three  for  position  and  three  for  the  magnitude  along  each  axis.  This  model 
does  not  however  represent  the  true  state  of  affairs  in  the  brain  since  there  are  multiple 
simultaneously  active  sources.  This  problem  is  dealt  with  by  using  magnetic 
encephalography  (MEG)  in  association  with  the  EEG  measurements. 

3.3  Eye  Movement  /  Position 
3.3.1  Electro-Oculogram  (EOG) 

The  EOG  is  a  measure  of  the  potential  difference  that  exists  between  the  cornea  and  the 
retina,  called  the  resting  or  standard  potential  of  the  eye.  The  cornea  is  approximately  6 
mV  positive  with  respect  to  the  retina,  which  changes  with  changing  retinal  illumination. 
The  standing  potential  of  the  eye  is  generated  mainly  by  the  transepithelial  potential 
across  the  pigmented  epithelium  of  the  retina. 

Thus  sensors  placed  on  the  right  and  left  external  canthi  of  the  eyes  (i.e.,  the  outside  or 
lateral  location,  just  next  to  each  eye)  will  pick  up  the  change  in  orientation  of  the  +/- 
dipole  of  each  eye.  Similarly,  sensors  placed  above  and  below  the  eye  will  provide 
corresponding  EOG  information  about  up  and  down  movement  of  the  eyeball.  In 
summary,  sensors  on  the  sides  of  the  eyes  provide  information  about  lateral  movement 
(X-axis)  and  sensors  placed  above  and  below  the  eyes  provide  information  about  up  and 
down  movement  (Y-axis).  In  combination,  then,  the  two  electrode  pairs  provide 
information  about  where  the  eyeball  is  oriented  in  an  X-Y  coordinate  system,  and  hence 
the  direction  of  gaze  (Figure  12). 
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Figure  12:  Typical  Placement  of  EOG  Sensors  for  Lateral  &  Vertical  Eye  Movements 

(From  McGuigan,  1979). 

Several  studies  have  been  performed  to  demonstrate  that  eye  behavior  changes  with 
alterations  in  workload,  task  difficulty,  fatigue,  and  the  nature  of  the  visual  display. 

Brown  and  Hufifinan  (1972)  have  demonstrated  that  lateral  eye  movements  are 
significantly  affected  in  driving  tasks.  They  foimd  that  the  highest  amoimt  of  visual 
activity  is  present  in  residential  driving,  followed  respectively  by  business  district, 
expressway,  and  rural  driving.  They  also  discovered  that  lateral  eye  movements  are  lower 
in  night  driving  as  compared  to  daylight  driving. 

Synchronization  of  the  horizontal  and  vertical  eye  movements  can  provide  the  point  of 
gaze  of  the  eyeballs,  and  this  information  can  be  overlaid  across  the  visual  display  that  the 
subject  views.  Such  processing  can  provide  secondary  visual  measurements  such  as  point 
of  fixation,  duration  of  fixation,  saccade  velocity,  and  the  patterns  of  visual  scanning 
across  a  display. 

Itoh  et  al.  (1990)  have  examined  the  characteristics  of  pilot  scanning  behavior  to  compare 
use  of  CRT  displays  versus  electromechanical  displays  in  the  cockpit.  They  observed  that 
the  altitude  indicator  is  the  most  frequently  gazed  at  instrument  within  the  cockpit. 

Studies  can  also  be  done  to  observe  and  compare  the  changes  in  behavior  in  normal 
versus  emergency  situations,  so  as  to  understand  how  attention  changes,  what  instruments 
are  viewed,  for  how  long,  and  in  what  order. 

Visual  scanning  behavior  has  also  been  used  as  a  way  of  comparing  the  skill  levels  of 
novices  and  experts.  It  has  been  found  by  Summala  et  al.  (1996)  that  novices  are  much 
more  affected  by  loading  tasks  than  experts  are,  and  rely  on  foveal  or  central  vision. 
Experts  on  the  other  hand,  leam  to  use  and  depend  upon  peripheral  vision,  and  are  able  to 
use  this  information  effectively  and  fast.  With  increasing  task  load,  the  scanning 
behavior  increases  in  novices,  while  the  average  dwell  time  or  the  duration  of  fixation 
increases  in  experts  (Tole  et  al.,  1982).  These  results  imply  that  monitoring  eye 
movements  can  be  used  as  an  indirect  but  effective  method  of  evaluating  and  assessing 
improvements  in  subjects  who  have  undergone  vigilance  training  for  target  search  and 
identification  tasks. 
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3.3.2  Blink  Measurement 

Several  blink  related  variables  such  as  blink  rate,  blink  amplitude,  inter  blink  interval, 
and  blink  closure  duration,  have  been  used  as  indicators  of  human  attention,  workload, 
vigilance,  and  fatigue. 

Both  eye  blink  rate  and  eye  blink  duration  are  useful  measures  of  workload  level. 
Research  has  shown  that: 

1 .  As  cognitive  workload  level  increases,  eye  blink  rate  decreases. 

2.  As  cognitive  workload  level  increases,  eye  blink  duration  decreases. 

It  appears  that  as  the  subject  is  forced  to  concentrate  harder,  they  focus  their  visual 
attention  along  with  their  mental  attention,  and  the  eyes  remain  open  more  as  a  result  of 
this  increased  intensity  and  focus. 

Hancock  et  al.  (1990)  have  demonstrated  the  relationship  between  blink  rate  and 
increased  workload  in  a  driving  task  in  curved  versus  straight  road  driving.  Eye  blink  rate 
is  lower  in  visually  demanding  and  high  workload  tasks.  Wilson  and  Fisher  (1991)  have 
demonstrated  this  effect  in  pilots  under  different  flight  segments,  and  used  blink  rate  to 
classify  mission  segments  using  discriminant  analysis. 

Blink  characteristics  in  conjunction  with  eye  movement  recordings,  could  therefore  be 
used  as  a  powerful  tool  for  estimating  cognitive  workload  and  operator  stress  and 
performance  in  specialized  tasks  and  environments. 

Several  studies  have  also  demonstrated  that  blink  rate  is  a  reliable  measure  of  fatigue 
associated  with  the  time-on-task  (TOT).  Gabon  et  al.  (1993)  have  shown  that  the  onset  of 
drowsiness  in  a  vigilance  type  task  is  marked  by  a  strong  increase  in  blink  rate  followed 
by  a  complete  disappearance  of  eye  blinks. 

Morris  and  Miller  (1996)  have  tested  specific  components  of  the  eye  and  eyelid 
movements  as  predictors  of  performance  decrements  resulting  from  fatigue  in  sleep 
deprived  pilots.  The  variables  that  they  measured  included  blink  rate  and  duration,  long 
closure  rate  (LCR),  blink  amplitude  (BL),  saccade  rate  and  velocity,  and  the  peak  saccade 
velocity.  Their  results  show  that  subjective  reports  of  fatigue  are  strongly  correlated  with 
increased  error,  and  that  blink  rate  is  the  best  predictor  of  error  performance,  followed  by 
blink  amplitude  and  long  closure  rate. 

As  an  alternative  to  the  EOG  or  Video  image  tracking  methods  for  measuring  eye 
movement,  the  blinkometer  device  (Figure  13)  may  be  used  to  strictly  measure  eye  blinks. 
Manufactured  by  IM  Systems,  located  in  Baltimore,  MD,  the  $800  device  consists  of  a 
sensor  that  mounts  near  the  eye  and  a  pager-sized  data-collection  unit  that  can  be  read  by 
the  lab  computer.  IM  Systems’  owner,  biomedical  researcher  David  T.  BCrausman,  says 
IM  plans  to  sell  a  $100  version  that  sounds  an  alarm  when  blinking  patterns  indicate  that 
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its  wearer  is  about  to  fall  asleep.  This  device  is  expected  to  find  applications  ranging 
from  truck  drivers  to  machine  tool  operators. 


Figure  13:  Blinkometer 


3.3.3  Infra-red  &  Video  Imaging  Based  Eye  Tracking 

The  EOG  measurement  discussed  previously  is  often  very  convenient  and  effective  when 
measuring  eye  motion,  but  is  typically  not  very  effective  at  determining  absolute  eye 
position  and  gaze  location  (i.e.  what  the  person  is  looking  at),  due  to  electrode  drift  over 
time.  Eye  position  and  gaze  location  can  be  more  effectively  determined  using  infra-red 
(IR)  sensing  and/or  video  imaging  techniques  that  measure  and  track  features  of  the  eye. 

IR  sensing  techniques  utilize  one  or  more  infra-red  point  sources  to  create  reflections  on 
the  eye;  these  reflections  are  then  measured  using  discrete  IR  sensors.  As  the  eye  moves, 
features  of  the  eyeball  change  the  reflection  characteristics  that  are  measured  by  the  IR 
sensors.  Calibration  and  mapping  algorithms  can  therefore  be  applied  to  determine  the 
position  of  the  eye  based  on  the  infra-red  measurements. 
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Several  research  and  commercial  eye  tracking  devices  have  used  this  type  of  approach.  It 
provides  for  a  relatively  low-cost  and  light-weight  implementation,  with  a  high  sampling 
rate.  As  a  disadvantage,  it  has  some  performance  limitations,  particularly  with  regards  to 
vertical  dimension  tracking,  and  it  may  be  difficult  to  setup  and  align  the  components. 

Alternately,  a  video  image  of  the  eye  may  be  used  to  measure  and  track  eye  position.  In 
this  case,  eye  position  is  determined  using  image  processing  algorithms  performed  on  the 
captured  image  of  the  eye.  These  algorithms  may  track  infra-red  reflections  (as  done  in 
the  previous  method),  or  detect  pupil  location  (using  infra-red  illumination  for  creating 
pupil-iris  contrast),  or  both.  Calibration  methods  are  then  used  to  map  pupil  center  and/or 
IR  reflection  locations  to  a  gaze  position. 

Video  based  eye  trackers  limit  gaze  determination  to  much  slower  sampling  rates  than 
can  be  achieved  using  the  discrete  IR  sensing  eye  trackers;  this  is  due  to  the  time  involved 
in  image  capture  and  processing.  However,  video  eye  trackers  generally  provide  better 
consistency  and  accuracy. 

With  both  methods  the  eye  tracking  system  may  be  either  head-moimted,  or  remote  from 
the  subject,  each  offering  different  advantages  and  disadvantages.  Remote  systems  can 
present  little  or  no  obstruction  to  the  subject’s  field  of  view,  but  require  keeping  head 
position  within  a  very  limited  space.  Head-mormted  systems  allow  for  free  head  motion, 
but  require  some  obstruction  of  the  subject’s  field  of  view  and  may  present  additional 
challenges  when  determining  gaze  (due  to  head  motion). 

Several  available  infra-red  sensing  (infra-red  oculography)  and  video  imaging  eye 
trackers  are  reviewed  in  Figure  14.  Cybernet  has  also  developed  our  own  custom  eye 
tracking  systems  based  on  video  imaging. 

Besides  the  significance  of  motion,  blink,  and  gaze  measurements  already  discussed, 
pupillary  responses  -  namely  the  size  of  the  pupil  -  have  also  been  used  as  an  indicator  of 
attention,  workload,  and  fatigue.  Peavler  (1974)  has  demonstrated  that  information 
overload  can  result  in  the  dilatation  of  the  pupil  due  to  the  mental  processing  effort  and 
the  involvement  of  task  related  anxiety.  Geacintov  and  Peavler  (1974)  have  used 
pupillography  in  industrial  fatigue  assessment  and  shown  that  pupil  constriction  occurs 
with  fatigue.  Pupil  size  can  be  readily  measured  using  video  based  eye  trackers  that 
already  use  pupil  detection  methods  to  determine  gaze  location. 
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position  and  pupil  arc.  Pupil  diameter:  20  degs  vertical 

size _  0.01  mm 
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3.4  Dermal  (Skin)  Measurements 

The  human  electroderaial  system  consists  of  the  eccrine  sweat  glands  concentrated  in  the 
digits  and  the  palms  of  the  hands.  It  is  iimervated  by  two  systems,  namely  the 
sympathetic  innervation  from  the  autonomic  nervous  system,  and  the  basal  ganglia- 
limbic-cortical  control  circuitry.  Electrodermal  activity  or  EDA  is  considered  one  of  the 
most  sensitive  indicators  of  psychological  activity.  EDA  has  been  measured  in  a  variety 
of  environmental  settings  like  work-sites,  traffic  jams  and  classrooms,  using  ambulatory 
recording  techniques  (Turpin,  1985). 

Sweat  gland  activity  is  responsible  for  EDA  that  is  recorded  as  changes  in  electrical 
resistance  of  the  skin.  It  is  hypothesized  that  the  increase  in  sympathetic  nervous  system 
activity  causes  an  increased  hydration  of  the  eccrine  sweat  glands  and  therefore  the 
surface  of  the  skin.  This  leads  to  a  drop  in  skin  resistance,  and  subsequently  an  increase 
in  skin  conductance  that  is  recorded  as  EDA.  EDA  is  therefore  an  indicator  of 
“psychological  sweating”  as  opposed  to  the  “thermoregulatory  sweating”,  and  occurs 
when  there  is  information  processing  and  cognitive  activity  in  the  human. 

3.4.1  Skin  Resistance,  Skin  Conductance  &  Spontaneous  Fluctuations 

Electrodermal  activity  can  be  measured  either  as  an  endosomatic  or  as  an  exosomatic 
activity.  Endosomatic  activity  refers  to  the  actions  potentials  generated  in  the 
sympathetic  nerves  under  the  skin.  A  technique  of  microneurography  developed  by 
Wallin  and  Fagius  (1986),  has  been  used  to  directly  measure  the  sympathetic  activity 
using  microelectrodes  inserted  into  the  nerve.  These  potentials  can  be  uniphasic  or 
biphasic,  depending  on  the  number  of  deflections  observed  in  the  recording.  Recordings 
have  shown  a  positive  correlation  between  the  bursts  in  sympathetic  activity  and  the 
amplitude  change  in  the  skin  resistance  measured.  Exosomatic  electrodermal  activity  is 
measured  directly  from  the  surface  of  the  skin.  A  small  current  is  passed  across  two 
electrodes  placed  on  the  skin  and  the  change  in  skin  resistance  is  recorded  as  a  function  of 
increased  sweat  gland  activity. 

EDA  can  record  two  types  of  activity  namely  the  slow  tonic  responses  occurring  in 
minutes,  and  the  fast  phasic  responses  occurring  within  seconds  following  the  stimulus. 
Phasic  changes  that  occur  in  the  absence  of  stimuli  are  known  as  spontaneous 
fluctuations.  An  example  of  a  recording  of  an  electrodermal  response  to  stimuli  is  shown 
in  Figure  15. 
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Figure  15:  Types  of  Electrodermal  Measures 


The  three  basic  modes  of  EDA  activity  are  closely  intercorrelated,  but  represent  different 
physiological  processes.  Tonic  activity  is  believed  to  reflect  vigilance,  sustained 
attention,  and  heightened  arousal  over  time  (Kilpatrick,  1972).  Phasic  responses  reflect 
both  the  cognitive  and  emotional  demands  of  a  stimulus,  especially  one  that  is  unexpected 
or  of  particular  significance  to  the  subject.  Barry  and  Sokolov  (1993)  have  foimd  that  the 
levels  of  both  tonic  and  phasic  responses  of  skin  conductance  decreases  as  a  function  of 
the  visual  stimulus  trials.  Spontaneous  fluctuations  show  high  sensitivity  to  arousal  and 
anxiety  by  increasing  in  their  frequency  of  occurrence  (Lader,  1967). 

Another  measure  of  electrodermal  activity  is  recovery  rate,  that  is  defined  as  the  time 
taken  by  electrodermal  activity  to  return  to  the  prestimulus  baseline  after  the  response 
peak.  In  reality,  it  is  measured  as  time  taken  by  the  recording  to  return  to  halfway  of  the 
baseline  (rec  t/2).  It  has  been  shown  by  Edelberg  (1970)  that  a  slow  recovery  correlates 
with  emotionality,  while  a  rapid  recovery  correlates  with  attention  and  goal-directed 
behavior. 

3.4.1. 1  Recording  and  Scoring  of  Electrodermal  Activity 

The  commonly  used  terminology  on  EDA  is  shown  in  Figure  16.  Tonic  changes  are 
expressed  as  skin  resistance  levels  or  skin  conductance  levels,  while  phasic  responses  are 
labeled  as  skin  resistance  responses  and  skin  conductance  responses. 
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Figure  16:  Recording  Parameters  for  Tonic  and  Phasic  Electrodermal  Activity 


Response  Type 

Resistance 

Conductance 

Tonic  level 

Skin  resistance  level 
(SRL,  KQ) 

Skin  conductance  level 
(SCL,pS) 

Phasic  response 
(elicited  or 
spontaneous) 

Skin  resistance  response 
(SRR,KQ) 

Spontaneous  skin 
resistance  fluctuations 
(SF,  Fluctuations  per 
minute) 

Skin  conductance 
response  (SCR,  pS) 

Spontaneous  skin 
conductance 
fluctuations  (SF, 
Fluctuations  per  minute) 

The  normal  range  for  tonic  changes  in  skin  conductance  is  1-30  fiS/cm^,  where  nS  stands 
for  microsiemens.  The  normal  range  of  the  amplitude  of  the  phasic  SCR  is  0.05  to  5  pS. 
An  important  measure  in  electrodermal  activity  is  the  smallest  accepted  phasic  response, 
which  is  set  to  0.05  pS,  but  can  be  as  low  as  0.01  pS  or  lower,  if  computerized  scoring 
programs  are  used. 

Strict  guidelines  have  been  laid  out  for  recording  of  electrodermal  activity  by  Fowles  et 
al.  in  1981.  In  particular,  the  sodium  chloride  (NaCl)  concentration  used  in  the  electrode 
paste  should  not  exceed  0.05  molar  concentration,  which  is  the  upper  limit  of  the  NaCl 
concentration  of  the  skin  surface  due  to  sweating.  It  is  recommended  that  a  non 
polarizable  silver/silver  chloride  electrode  be  used  together  with  an  electrode  paste  of 
0.05  molar  NaCl,  made  from  a  mix  of  unibase  paste  and  3%  saline  solution. 
Electrodermal  responses  can  be  elicited  from  several  sites  such  as  the  volar  or  dorsal 
aspect  of  the  hand,  the  plantar  aspect  of  the  foot,  the  ankle,  the  forehead,  or  the  abductor 
hallucis  muscle.  Care  should  be  taken  not  to  clean  the  electrode  site  with  alcohol  or 
abrasive  cleaning  paste  since  this  can  disrupt  the  functioning  of  the  sweat  glands. 

Phasic  responses  of  EDA  can  be  scored  in  several  ways,  some  of  which  are  illustrated  in 
Figure  17. 
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Stimulus!  Stimulus  2  Stimulus'3 


Figure  1 7;  Parameters  of  a  Phasic  Electrodermal  Response 


A  response  is  measured  from  a  deflection  point  to  the  response  peak.  Averaging 
responses  across  trials  including  “zero  responses”,  estimates  the  stimulus  response 
magnitude,  while  averaging  without  the  “zero  responses”  estimates  the  mean  response 
amplitude.  Response  frequency  is  the  number  of  responses  per  imit  time,  and  response 
probability  is  the  number  of  identifiable  responses  in  proportion  to  the  number  of  trials. 
Recovery  rate  is  measured  as  the  time  it  takes  for  the  recording  to  return  from  the  peak 
value  to  half  the  displacement  from  the  baseline  and  is  called  the  rec  t/2,  indicating  the 
time  to  50%  recovery.  Another  way  of  representing  recovery  time  can  be  rec  t/4  which 
refers  to  the  time  to  25%  recovery.  Rise  time  is  the  time  to  reach  the  peak  (maximum 
amplitude)  from  the  onset  of  the  stimuli  and  its  value  ranges  between  1  and  4  seconds. 
Latency  refers  to  the  time  of  onset  of  a  stimulus  to  the  initiation  of  a  response,  with  a 
value  of  1-2  seconds  for  skin  conductance  responses. 

In  certain  instances,  multiple  responses  may  occur  within  the  scoring  “window”.  It  has 
been  observed  by  Lockhart  (1966)  that  in  experiments  with  two  stimuli  and  long 
interstimulus  intervals  between  the  two  stimuli,  three  distinct  responses  are  present  as 
shown  in  Figure  18.  They  have  been  named  as  the  first  anticipatory  response  (FAR),  the 
second  anticipatory  response  (SAR),  and  the  third  omission  response  (TOR).  The  FAR 
has  been  attributed  to  perceptual  and  attentional  processes  like  the  orienting  response,  the 
SAR  reflects  a  conditioned  response,  in  anticipation  of  the  strong  unconditioned  stimulus 
(UCS),  and  the  TOR  reflects  learning,  referring  to  the  omission  on  test  trials  with  the 
UCS. 
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TOR 


Conditioned  Stimulus  (>5sec) 


Figure  18:  Example  of  FOR,  SAR,  and  TOR  in  an  Experiment  Using  a  Conditional 

Stimulus 


3.4.1 .2  Cognitive  Implications  of  EDA 

The  mechanisms  of  control  of  electrodermal  phenomena  from  the  highest  centers  are  yet 
to  be  completely  understood.  It  has  been  observed  that  areas  in  the  cortex  like  the  lateral 
frontal  cortex  and  the  dorsolateral  cortex  have  direct  influence  on  the  electrodermal 
response.  Neuroanatomical  correlates  of  electrodermal  responses  have  shown  that 
cortical  area  activity  is  critical  in  the  elicitation  of  skin  conductance  response  (Tranel  and 
Damasio,  1994).  From  a  functional  viewpoint,  the  central  nervous  system  control  of 
EDA  can  be  examined  as  three  separate  systems,  namely  the  locomotor,  orienting- 
activating  and  the  thermoregulatory  systems. 

Furedy  (1993)  has  succinctly  discussed  the  notion  of  electrodermal  activity  as  a  tool  to 
differentiate  psychological  processes  in  humans.  He  emphasizes  the  physiological 
differentiation  between  the  sympathetic  and  the  parasympathetic  nervous  system  that  may 
be  relevant  in  drawing  the  psychological  distinction  between  stimuli  that  result  in  an 
increase  in  stress  from  those  that  result  merely  in  an  increase  in  attention.  An  example  to 
illustrate  this  importance  of  such  a  distinction,  is  the  need  to  differential  the  indicators  of 
susceptibility  to  cardiac  arrest  from  those  stress  indicators  due  to  increased  attention. 
Since  electrodermal  activity  is  known  to  predominantly  reflect  sympathetic  nervous 
system  activity,  an  increase  in  electrodermal  level  can  be  reasonably  used  as  an  index  of 
stress.  In  contrast,  preparation  for  a  stressful  task  where  more  attentional  resources  are 
demanded  than  the  development  of  stress  per  say,  will  cause  an  acceleration  in  heart  rate 
without  a  change  in  the  electrodermal  level.  Another  example  is  the  electrodermal 
differentiation-of-deception  (DDP)  where  falsely  or  deceptively  answered  questions  cause 
a  larger  skin  conductance  response  than  honestly  answered  questions  (Vincent  and 
Furedy,  1992),  and  this  principle  has  been  used  for  many  years  on  polygraphy. 
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The  potential  advantage  of  investigating  the  psyche  using  the  low  technology  tool  of 
electrodermal  activity  cannot  be  sufficiently  emphasized.  It  has  been  observed  that 
involuntary  physiological  changes  detected  through  electrodermal  activity  can  be  used  to 
explore  the  emotional  and  cognitive  aspects  of  the  psyche  more  easily  and  effectively 
than  methods  of  greater  technical  sophistication. 

3.4.1. 3  Commercial  Application:  MindDrive™  Technology 

One  such  commercial  tool  using  the  concept  of  electrodermal  activity  is  a  computer  game 
called  MindDrive"^”.  It  works  on  the  principle  that  differing  mental  activities  like 
remembering,  relaxing,  analyzing,  positive  and  negative  thoughts  and  left  and  right  brain 
activity,  all  generate  different  bioelectric  patterns  that  are  measured  through  the  skin. 
Users  can  volitionally  control  cursor  movement  by  getting  into  an  “excitatory”  state  or 
into  a  “relaxed”  state  of  mind.  This  concept  is  illustrated  in  Figure  19. 


Figure  19:  Cursor  Control  Using  Thought  Processes  in  MindDrive 


The  sensor  that  is  used  is  a  finger  sensor  which  measures  heart  rate,  temperature,  blood 
pulse  volume,  and  Composite  Neural  Activity  (CNA)  which  is  composed  of  a  set  of 
multiple  bioelectric  signals.  The  MindDrive  technology  automatically  and  continuously 
analyzes  the  following  components  of  CNA  in  the  following  manner: 

1 .  Amplitude  (output  and  strength  of  the  signals) 

2.  Velocity  (speed  or  rate  at  which  the  signals  change) 

3 .  Attenuation  (size  of  the  changes) 

4.  Delta  or  asymmetiy  of  the  CNA  signals,  as  compared  to  each  other 

5.  Combination  of  both  the  phasic  (rate  of  change)  and  tonic  (absolute  measurement) 
signals. 

A  picture  of  the  finger  sensor  is  shown  in  Figure  20. 
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Figure  20:  Finger  Sensor  Used  in  MindDrive 


The  two  major  obstacles  that  have  prevented  the  use  of  electrodermal  signals  to  control 
devices  and  computers  are: 

1 .  The  2.5  to  3.5  second  delay  between  a  human  thought  and  the  traditional 
measurement  of  the  thought  response  at  the  skin. 

1 .  Inability  to  differentiate  the  changes  generated  by  cognitive  volitional  thoughts 
from  those  resulting  out  of  the  influence  of  the  autonomic  nervous  system  on 
heart  beat,  blood  pulse  volume,  and  temperature. 

It  is  claimed  that  the  MindDrive  technology  is  able  to  decipher  the  differences  between  i 
physiological  signal  generated  by  volitional  thought  (transmitted  though  the  central 
nervous  system)  and  one  that  has  been  changed  by  the  autonomic  nervous  system.  This 
difference  is  recognized  through  a  filtering  process  of  the  changes  generated  by  the 
autonomic  nervous  system,  so  that  only  the  signals  generated  by  cognitive/volitional 
thoughts  remain.  This  system  was  tested  at  Cybernet  Systems  Corporation  and  it  was 
found  that  subjects  could  control  the  cursor  to  a  certain  degree.  A  representation  of  this 
system  is  shown  in  Figure  21. 
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Figure  21:  User  Controlling  a  Game  Through  MindDrive  Technology 


Electrodermal  activity  shows  promising  value  in  application  for  the  assessment  of 
cognitive  human  performance. 

3.4.2  Sweat  Rate 

Eccrine  sweat  glands  are  controlled  by  the  sympathetic  nervous  system,  and  primarily 
responsible  for  electrodermal  activity.  These  are  distributed  all  over  the  human  body,  but 
occurring  with  a  greater  density  in  the  palms  of  the  hands  and  the  soles  of  the  feet.  The 
density  in  these  regions  ranges  around  1000  glands/cm^ ,  as  compared  to  100-200  glands/ 
cm^.  Besides  the  influence  of  the  sympathetic  branch  of  the  autonomic  nervous  system, 
sweat  glands  are  also  influenced  by  circulating  epinephrine,  indicating  a  hormonal 
influence  on  electrodermal  response. 

An  important  factor  in  determining  the  magnitude  of  the  electrodermal  response  is  the 
amount  of  perspiration  or  the  level  of  hydration  of  the  sweat  gland.  A  simple  and 
accurate  test  to  measure  sweating  is  the  use  of  the  palmar  sweat  index  (PSI)  developed  by 
Sutarman  and  Thomson  (1952).  The  PSI  has  been  shown  to  correlate  significantly  with 
both  the  spontaneous  fluctuations  as  well  as  the  skin  conductance  responses.  The  PSI  is  a 
count  of  the  active  glands  in  a  specific  area  of  the  skin.  A  plastic  impression  of  a  patch  of 
skin  on  a  fingertip  is  made  after  the  presentation  of  a  stimulus,  or  over  time,  and  the 
number  of  “sweat  dots”  is  a  measure  of  the  electrodermal  activity  to  a  psychologically 
relevant  event.  The  palmar  sweat  index  is  a  sensitive  measure  of  both  anxiety  and 
arousal,  and  can  be  used  with  ease  in  situations  where  conventional  recordings  are 
difficult  to  perform. 
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3.5  Cardiovascular  Measurements 

Another  organ  that  comes  under  the  influence  of  the  autonomic  nervous  system  is  the 
heart.  Two  specialized  regions  in  the  heart,  namely  the  sino-atrial  node  and  the  atrio¬ 
ventricular  node,  are  responsible  for  regulating  the  heart  beat,  with  the  sino-atrial  node 
being  the  pacemaker.  Both  the  SA  and  AV  nodes  are  innervated  by  the  vagus  nerve  from 
the  parasympathetic  nervous  system.  This  has  an  inhibitory  effect  on  the  heart,  and 
decreases  the  heart  rate.  On  the  other  hand,  the  sympathetic  nervous  system  has  an 
excitatory  effect  on  the  heart  causing  an  increase  in  the  frequency. 

3.5.1  Heart  Rate,  Stroke  Volume,  and  Inter-beat  Interval 
Important  concepts  of  heart  physiology  are  the  heart  rate,  stroke  volume  and  interbeat 
interval.  Heart  rate  is  the  number  of  times  the  heart  beats  per  minute  (bpm).  Stroke 
volume  is  the  amount  of  blood  ejected  at  each  beat  of  the  heart.  Both  these  parameters 
are  related  to  a  third  parameter,  cardiac  output  which  is  the  quantity  of  blood  pumped  by 
the  ventricle  per  minute  in  the  following  manner. 

Cardiac  Output  (liters)  =  Stroke  Volume  (litersAipm)  x  Heart  Rate  (bpm) 

The  heart  rate  can  also  be  expressed  as  another  parameter,  namely  the  inter-beat  interval. 
This  is  the  time  between  two  heart  beats  in  milliseconds  and  is  determined  from  the  ECO 
recording.  The  heart  rate  is  related  to  the  inter-beat  interval  by  the  following  relationship. 

Heart  Rate  (bpm)  =  (60  x  10^)  /  Inter  -beat  Interval  (msec) 

All  the  above  parameters  are  regulated  both  by  nervous  system  control  as  well  as 
hormonal  control. 

Heart  rate  activity  can  be  recorded  using  the  electro-cardiograph.  This  records  the 
electrical  currents  generated  by  the  action  potentials  of  the  cardiac  muscle  cells  through 
electrodes  placed  on  the  skin.  A  typical  ECG  recording  is  shown  in  Figure  22.  The  ECG 
recording  consists  of  a  P  wave,  a  QRS  complex  and  a  T  wave. 


R 


Figure  22:  Typical  Electrocardiogram  Recording 
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The  heart  rate  or  the  frequency  with  which  the  heart  beats  can  be  determined  using  the 
time  in  milliseconds  between  two  consecutive  R  waves,. also  called  the  inter-beat  interval. 
There  is  a  reciprocal  relationship  between  the  heart  rate  and  the  inter-beat  interval.  In 
psychophysiological  research,  measurement  of  the  changes  in  heart  rate  as  a  fimction  of 
stimulus  presentation  are  based  on  identifying  successive  R  waves  in  the  ECG.  Choice  of 
using  the  heart  rate  versus  the  inter-beat  interval  (or  heart  period)  is  dependent  on  the  type 
of  study  performed.  In  many  cases  the  inter-beat  interval  is  used  as  the  dependent 
measure.  Figure  23  shows  the  appearance  of  an  inter-beat  interval  plot. 


Figure  23:  Inter-beat  Interval  Plot  Showing  the  Beat-to-Beat  Fluctuation 

Heart  rate  was  one  of  the  earliest  physiological  parameters  used  to  monitor  the  state  of  the 
human  operator,  with  both  physical  and  mental  activity  being  known  to  increase  the  heart 
rate.  Many  studies  have  observed  systematic  relations  between  cognitive  demands  and 
heart  rate  in  different  environments  (BCramer,  1991;  Roscoe,  1992;  Wilson  and  Eggemier, 
1991).  Several  studies  have  also  looked  at  heart  rate  characteristics  during  aircraft 
piloting:  flying  combat  missions  (Lewis  et  al.,  1967),  flying  surface  attack  training 
missions  (Wilson,  1993),  flying  aircraft  test  missions  (Roscoe,  1980),  gradient  of  landing 
approach  (Roscoe,  1975),  pilot  versus  copilot  flying  (Hart  and  Hansen,  1987;  Kakimoto 
et  al.,  1988,  Roscoe,  1978),  and  high  speed  low  level  flight  (Lidderdale,  1987).  All  these 
studies  have  shown  that  with  an  increase  in  task  difficulty,  there  is  an  increase  in  heart 
rate.  Rokicki  (1987)  and  Roscoe  and  Ellis  (1990)  have  used  heart  rate  as  a  debriefing  tool 
for  aircraft  test  and  evaluation,  as  well  as  in  the  certification  of  commercial  aircraft 
(Roscoe,  1987;  Speyer  et.  al.,  1988;  Wainwright,  1988). 

All  the  above  studies  have  demonstrated  an  increase  in  heart  rate  with  mental  workload. 

A  study  by  Wilson  and  Fisher  (1991)  used  cardiac  variables  and  eye-blink  to  classify 
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flight  segments  in  pilots  and  systems  officers  for  air-to-ground  training  and  weapon 
sessions.  Similar  results  have  been  reported  in  studies  relating  to  automobile  driving 
(Helander,  1975),  boat  driving  (Johnson,  1980)  and  in  radar  operation  (O’Hanlon  and 
Beatty,  1977). 

Changes  in  heart  rate  as  a  function  of  attention  and  vigilance  has  also  been  observed  in 
several  psychophysiological  studies.  Studies  on  heart  activity  following  a  stimulus  have 
shown  an  initial  cardiac  deceleration  that  is  sometimes  followed  by  an  acceleration,  and  a 
second  deceleration.  Cardiovascular  psychophysiology  has  been  explained  in  terms  of 
information  processing  using  the  heart  rate  curve.  A  typical  triphasic  heart  rate  curve  is 
illustrated  in  Figure  24.  An  initial  deceleration  D1  is  associated  with  the  cognitive 
processes  of  focusing  attention  and  orienting  to  the  stimulus.  The  acceleration  A  has 
been  attributed  to  the  emotional  aspects  of  the  stimulus,  or  to  the  internal  cognitive 
processing.  The  second  deceleration  D2  is  attributed  to  tibe  anticipation  of  the  next 
stimulus. 


8  sec 

SI  _ ^  S2 

Ready  Press 


Figure  24:  A  Typical  Tri-phasic  Heart  Rate  Curve  Shown  Against  Stimulus  Presentation 

Venables  (1991)  proposed  that  the  deceleration  of  the  heart  rate  is  coupled  to  an  “open 
attentional  stance”,  and  cardiac  acceleration  to  a  “closed  attentional  stance”,  both  of 
which  are  related  to  sensory  intake  and  environmental  rejection  respectively. 

3.5.2  Heart  Rate  Variability 

Heart  rate  variability  has  also  been  used  as  a  measure  for  evaluating  task  load.  Many 
studies  reported  a  decrease  in  heart  rate  with  increasing  task  demands. 
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Heart  rate  power  spectrum  analysis  (HRPS  A)  is  a  relatively  new  technique,  based  on 
analysis  of  the  R-to-R  interval,  to  determine  the  heart  rate  variability  or  oscillations. 

Heart  rate  oscillations  have  been  shown  to  fall  into  three  spectral  peak  categories:  a  high 
frequency  component  peaking  around  0.2  to  0.4  Hz,  a  mid  frequency  component  peaking 
around  0.05  to  0.10  Hz,  and  a  low  frequency  component  peaking  at  0.04  Hz.  The  high 
frequency  component  has  been  found  to  be  sensitive  to  vagal  influences,  the  mid 
frequency  to  be  a  quantitative  measure  of  the  sympathetic  nervous  system  (Pagani  et  al., 
1986),  and  the  low  frequency  component  being  an  indicator  of  both  sympathetic  and 
parasympathetic  activity.  Figure  25  illustrates  the  different  frequent  components  in  a  heart 
rate  power  spectrum  analysis. 


Frequency  in  Hz 


Figure  25:  Power  Density  Spectrum  of  Heart  Rate  with  Peaks  in  Low,  Middle  and  High 

Frequency  Ranges 

HRPSA  can  display  the  variability  in  the  heart  rate  signal  as  a  power  density  spectrum 
with  Fourier  analysis.  Specifically  it  has  been  observed  that  the  mid  frequency  band, 
namely  the  0.10  Hz  component,  is  sensitive  to  the  amount  of  mental  effort  required  by  a 
task  (Mulder,  1992),  and  that  a  reduction  in  the  0.10  Hz  component  is  an  evidence  of 
resource  limited  tasks.  However  increased  task  difficulty  in  data-limited  tasks  are  not 
associated  with  a  reduction  in  the  0.10  Hz  component. 

Opmeer  and  Krol  (1973)  have  reported  that  the  heart  rate  variability  and  respiration  are 
sensitive  to  simulated  flight  task  demands.  Itoh  et  al.,  (1989)  found  the  heart  rate 
variability  in  the  0.10  Hz  band  to  decrease  during  take  off  and  landing  when  compared  to 
the  flight  cruise  segments.  Transient  air  craft  difficulties  have  been  found  to  produce  a 
decreased  0.10  Hz  activity.  Wilson  (1993)  found  a  decrease  in  the  mid  and  high  frequent 
band  activity  during  high  demand  segments  of  air-to-ground  missions  and  during  a 
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ground  based  tracking  task.  However  neither  band  could  distinguish  between  flight 
segments  or  between  the  tracking  tasks. 


Egelund  (1982)  reported  changes  in  heart  rate  variability  as  a  function  of  driving 
conditions  and  driver  fatigue.  Joma  and  Mulder  (1983)  found  differences  in  heart  rate 
variability  between  novice  and  expert  underwater  divers.  It  is  believed  that  the  high 
frequency  band  reflects  respiratory  activity  and  provides  an  index  on  the  influence  on  the 
heart  by  the  parasympathetic  nervous  system.  This  band  width  has  been  found  to  be 
sensitive  to  manipulations  of  cognitive  workload  (Forges  and  Byrne,  1992). 

3.5.3  T  wave  Characteristics  and  Chaos  Theory  in  Heart  .Function 

The  T  wave  amplitude  of  die  ECG  recording  was  initially  suggested  to  be  an  index  of 
sympathetic  activity  (Heslegrave  and  Furedy,  1980),  but  this  assumption  was  questioned 
leading  to  the  non  use  of  this  index.  Instead,  a  different  measure  called  the  T-dterans 
was  proposed  and  is  used  to  analyze  the  repetitive  pattern  of  T  wave  amplitude  (Verrier 
and  Nearing,  1994).  The  T  wave  alterans  has  been  suggested  to  be  an  important  index  of 
vulnerability  to  cardiac  failure. 

The  most  recent  trend  in  heart  physiology  is  the  application  of  non  linear  dynamics  and 
chaos  theory  to  analyze  heart  rate  and  frequency  (Goldberger,  1991).  This  method  has 
been  championed  for  the  detection  of  imminent  cardiovascular  dysfunction,  especially 
arrhythmias.  It  has  been  observed  that  the  heart  rate  of  a  patient  before  sudden  death  due 
to  cardiac  arrhythmia  is  periodic  and  non  chaotic,  whereas  the  variability  of  the  normal 
heart  follows  algorithms  predicted  from  fractal  and  chaos  theory. 

The  importance  of  chaos  theory  and  non  linear  dynamics  of  the  heart  rate  on  human 
performance  and  operator  state,  and  their  relevance  to  psychophysiology,  has  yet  to  be 
understood. 

3.5.4  Blood  Pressure,  Pulse  Transit  Time,  and  Pulse  Wave  Velocity 

The  flow  of  blood  through  a  vessel  is  determined  by  the  pressure  exerted  by  the  flow 
divided  by  the  resistance  of  the  vessel  to  the  passing  blood.  The  arterial  pressure  is 
determined  by  the  product  of  cardiac  output  and  the  total  peripheral  resistance.  The 
arterial  pressure  refers  to  both  the  systolic  and  the  diastolic  pressures,  which  are  the 
highest  and  the  lowest  pressures  in  the  cardiac  cycle.  The  systolic  varies  between  120 
and  140  mm  Hg  and  the  diastolic  varies  between  80  and  90  mm  Hg.  The  mean  arterial 
pressure  (MAP)  is  an  important  measure  of  the  functional  status  of  the  cardiovascular 
system,  and  is  defined  as  the  difference  between  the  diastolic  and  systolic  pressures  plus 
1/3  of  the  diastolic  pressure.  With  relation  to  mental  activity,  studies  have  observed  that 
shifts  in  the  EEG  in  the  frontal  cortex  area  were  smaller  when  baroreceptor  stimulation 
occurred  than  when  baroreceptor  inhibition  occurred. 

Pulse  Transit  Time  (PTT)  and  the  carotid  dP/dt  are  two  measures  of  the  of  the  rate  of 
change  in  blood  pressure  when  the  blood  is  ejected  from  the  left  ventricle.  Both  these 
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methods  measure  the  time  it  takes  for  the  pulse  from  the  peak  of  the  R  wave  to  reach  a 
certain  point  at  a  distance  from  the  heart.  Carotid  dP/dt  .is  a  measure  of  the  relative 
change  in  pressure  (dP)  in  relation  to  a  change  in  time  (dt).  Pulse  transit  time  is  measured 
in  present  times  using  impedance  techniques  while  carotid  dP/dt  is  measured  using  a 
piezo-sensitive  device  that  is  present  in  a  neck  collar  that  the  subject  wears. 

Pulse  wave  velocity  is  a  measure  of  the  speed  of  the  wave  traveling  along  a  viscoelastic 
wall  of  a  blood  filled  vessel  using  a  photo-plethysmographic  sensor  (Jennings  and  Choi, 
1983).  It  is  closely  related  to  the  pulse  transit  time.  Both  the  pulse  transit  time  and  the 
pulse  wave  velocity  are  considered  as  die  index  of  sympathetic  influences  on  the  heart. 
Increase  in  the  sympathetic  influence  on  the  heart  will  result  in  increased  contractility  and 
stroke  volume,  resulting  in  a  decreased  time  for  the  pulse  to  reach  the  sensing  device  on 
the  carotid  artery.  Pulse  transit  time  has  been  demonstrated  to  covary  with  the  systolic 
blood  pressure  under  different  stressful  conditions  (Obrist  et  al.,  1979). 

3.5.5  Peripheral  Pulse  Volume 

The  volume  of  the  blood  flowing  through  blood  vessels  in  the  periphery  can  be  measured 
to  understand  the  cardiovascular  state.  Both  phasic  and  tonic  changes  can  be  measured  in 
the  peripheral  blood  flow  as  pulse  volume  or  pulse  amplitude  and  as  blood  volume 
respectively.  Phasic  changes  are  the  beat-to-beat  fluctuations  that  occur  for  each  stroke  of 
the  heart.  Tonic  changes  occur  over  longer  periods  of  time.  Figure  26  shows  the  changes 
observed  in  blood  volume,  pulse  amplitude  and  blood  flow  measured  from  a  digit  during 
the  presentations  of  three  auditory  stimuli.  The  increase  in  pulse  amplitude  indexed  by 
the  vasodilatation  of  the  peripheral  blood  vessels,  is  caused  by  the  sympathetic  nervous 
system.  A  decrease  in  pulse  amplitude  is  caused  by  increased  sympathetic  activity  that 
results  in  vasoconstriction  of  the  peripheral  blood  vessels. 
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Figure  26:  Changes  in  Blood  Volume,  Pulse  Amplitude  and  Blood  Flow  During  Stimulus 

Presentations 
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Some  studies  have  been  performed  to  demonstrate  the  relationship  between  vascular 
parameters  and  human  behavior.  Sokolov  (1963)  used  plethysmographic  recordings  of 
the  fingers  and  forehead  to  distinguish  between  orienting  (OR)  and  defensive  responses 
(DR)  to  sensory  stimuli.  He  found  that  at  the  fingers,  both  the  OR  and  the  DR  result  in 
vasoconstriction  due  to  sympathetic  activity,  while  at  the  forehead  OR  results  in 
vasodilatation  and  DR  in  vasoconstriction,  since  there  is  both  a  sympathetic  and 
parasympathetic  influence. 

There  are  several  methods  for  measuring  vasoconstriction  and  vasodilatation,  by 
measuring  the  changes  in  the  peripheral  blood  volume.  A  standard  method  is  by  the  use 
of  plethysmographs,  which  record  the  change  in  blood  flow.  There  are  several  types  of 
plethysmography  depending  on  the  type  of  sensor  used.  Photoplethysmography  utilizes  a 
light  emitting  diode  with  a  sensor  on  either  side  of,  for  example,  the  thumb.  The  amount 
of  light  reflected  to  the  sensor  device  is  dependent  on  the  amount  of  blood  flowing  in  the 
peripheral  vessel.  For  example,  as  more  blood  flows,  less  light  gets  reflected. 

Other  types  of  plethysmography  are  impedance  plethysmography  and  Doppler  flowmetry. 
Impedance  plethysmography  measures  the  changes  in  blood  volume  by  measuring  the 
changes  in  electrical  resistance.  Doppler  flowmetry  on  the  other  hand,  measures  blood 
flow  velocity  by  acoustical  monitoring. 

3.6  Respiratory  Measurements 

Respiratory  psychophysiology  has  had  little  impact  on  main  stream  psychophysiology  in 
the  last  decades.  However  respiration  measurements  have  found  a  place  in  several  types 
of  studies  such  as: 

1 .  To  evaluate  the  effects  of  work  demands, 

1 .  To  evaluate  the  effects  of  environmental  stressors  such  as  G  forces  and  vestibular 
stimulation, 

1 .  To  detect  hyperventilation  due  to  over-reactivity  caused  due  to  danger  or  threat, 
and 

1 .  To  obtain  an  approximation  of  energy  expenditure. 

One  reason  for  the  poor  popularity  of  respiration  is  the  complex  nature  of  breathing 
patterns  which  vary  both  in  time  and  volume.  Noninvasive  measurement  techniques  only 
provide  an  assessment  of  respiratory  rate  and  not  of  volume.  Respiratory  rate  is  quite 
insensitive  to  workload  and  stress,  and  does  not  provide  meaningful  information.  It  is 
currently  accepted  that  respiratory  measures  can  be  useful  only  when  both  volume  and 
time  parameters  are  simultaneously  measured. 

3.6.1  Breathing  Rate  and  Breathing  Volume 

The  respiratory  cycle  is  composed  of  several  parameters.  These  are  the  duration  of 
inspiration  (Ti),  duration  of  expiration  (Te),  total  cycle  duration  (Ttot),  respiratory  rate 
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(inverse  of  Ttot),  and  tidal  volume  (VT)  which  is  the  volume  displaced  in  one  breath. 
Figure  27  shows  the  components  of  a  single  ventilatory  cycle. 


Figure  27:  Decomposition  of  Minute  Ventilation  into  Components  of  Central 

Respiratory  Regulation 

Two  fundamental  mechanisms  control  the  breathing  cycle.  These  are  the  drive 
mechanism  regulated  by  chemoreceptors  which  controls  the  firing  rate  of  the  inspiratory 
neurons,  and  the  timing  mechanism  regulated  by  the  central  rhythm  generator,  which 
switches  these  neuron  between  the  on  and  off  positions.  These  mechanisms  can  further 
be  described  by  combination  parameters  such  as  the  inspiratory  flow  rate  (VT/  Tj)  for  the 
drive  mechanism,  and  the  duty  cycle  time  (Tj/Ttot)  for  the  timing  mechanism.  Another 
parameter  called  the  minute  volume,  which  is  the  total  volume  ventilated  per  minute,  is 
equal  to  the  product  of  the  inspiratory  flow  rate  and  the  duty  cycle  time. 

Respiratory  time  and  volume  parameters  can  be  measured  non-invasively  using  the 
technique  of  inductive  plethysmography.  This  method  uses  techniques  to  measure  the 
motion  of  the  rib  cage  and  the  abdomen,  and  does  not  interfere  with  normal  breathing. 
Respiratory  volume  changes  are  measured  based  on  a  model  that  describes  the  chest  wall 
as  a  system  moving  with  2  degrees  of  freedom.  After  proper  calibration  of  the  system, 
measurement  of  the  changes  in  the  rib  cage  and  chest  wall  circumference  can  provide 
reasonably  accurate  measurements  of  volume  and  time  components. 

Innumerable  studies  have  been  done  in  the  past  to  assess  the  changes  in  respiration  with 
performance  of  effort  demanding  tasks  and  stressors.  With  mild  mental  effort  and  stress, 
an  increase  in  respiratory  rate,  decrease  in  tidal  volume  and  an  increase  in  minute  volume 
have  been  observed  (Turner  and  Carroll,  1985;  Wientjes,  1992).  However  with  extreme 


49 


CYBERNET  SYSTEMS  CORPORATION 


workload  demands  (Allen  and  Crowell,  1989)  and  with  painful  passive  coping  stress 
(Allen,  Sherwood  and  Obrist,  1986),  an  increase  in  tidal  volume  has  been  observed.  The 
use  of  respiratory  rate  as  a  measure  has  been  questioned  since  increase  in  respiratory  rate 
with  workload  is  confounded  with  other  factors  like  vocalization,  motor  activity, 
aversiveness,  etc. 

It  is  currently  believed  that  the  drive  mechanism  is  sensitive  to  work  demands. 

Inspiratory  flow  rate  was  found  to  discriminate  reliably  between  task  demands  in  the 
performance  of  a  memory  comparison  task  (Wientjes,  1992),  implying  that  a  greater 
workload  is  associated  with  a  stronger  inspiratory  drive.  However  the  timing  mechanism 
was  not  seen  to  be  influenced  by  workload. 

3.6.2  End-tidal  pC02  and  Transcutaneous  pC02 

Since  respiratory  changes  are  also  influenced  by  changes  in  metabolic  processes,  it  can 
also  be  important  to  measure  indices  of  the  metabolic  process.  One  such  index  is  the  end- 
tidal  pCOa.  This  is  measured  using  the  capnograph  which  is  attached  to  the  subject’s 
face-mask.  A  reduction  in  the  end-tidal  pC02  is  indicative  of  hyperventilation  which  is 
ventilation  in  excess  of  the  metabolic  requirements.  Since  the  use  of  a  face-mask  is 
relatively  cumbersome,  newer  techniques  such  as  transcutaneous  pC02  measurement 
methods  have  become  available.  This  method  uses  a  heated  electrochemical  sensor  that 
measures  the  arterial  pC02  through  the  skin.  The  output  voltage  is  proportional  to  the 
logarithm  of  the  CO2  concentration  of  tiie  skin.  The  sensor  is  heated  to  about  45  degrees 
Celsius  which  causes  a  vasodilatation  of  the  capillaries  under  the  skin,  and  arterialization 
of  the  capillary  blood.  Therefore  the  CO2  concentration  in  the  skin  tissue  is  assumed  to 
be  representative  of  the  arterial  pC02.  The  main  disadvantage  of  this  method  is  the  time 
lag  of  the  response  of  the  sensor  to  the  change  in  arterial  pC02.  The  transcutaneous 
measurement  of  pC02  has  been  used  in  studies  involving  ambulatory  monitoring 
(Pilsbury  and  Hibbert,  1987). 

Few  studies  have  used  the  end-tidal  pC02  levels  in  assessing  its  change  with  mental  load. 
It  has  been  demonstrated  that  in  active  coping,  where  the  subject  is  actively  involved  in 
performing  a  demanding  mental  task,  there  is  no  associated  hyperventilation  or 
hypocapnia.  It  is  therefore  believed  that  the  ventilatory  changes  observed  in  these  tasks 
are  mainly  due  to  the  changes  in  metabolic  activity  brought  about  by  motor  activity. 

Hyperventilation  can  also  be  brought  on  by  certain  aversive  conditions  such  as  pain  and 
cold  pressor  tasks.  This  can  have  an  important  effect  on  the  subjective  and  objective  state 
of  the  human,  both  due  to  the  manifestation  of  adverse  somatic  reactions  and  a  reduction 
in  myocardial  and  cerebral  perfusion  and  therefore  oxygen  delivery.  The  latter  can  lead  to 
a  decrement  in  cognitive  functioning  (Grossman  and  Wientjes,  1989). 

Some  applied  studies  have  used  respiratory  measures  to  characterize  human  performance 
and  operator  state,  as  in  the  study  by  Harding  (1987)  with  jet  fighter  pilots  in  different 
flight  segments.  Results  indicated  that  respiratory  rate,  peak  inspiratory  flow,  and  minute 
volume  were  increased  during  all  phases  of  the  regular  flight,  but  particularly  elevated 
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during  highly  demanding  maneuvering  phases.  It  has  been  suggested  that  stress  in  flight 
can  cause  a  significant  reduction  in  pC02  leading  to  detrimental  effects  in  psychomotor 
performance  (Gibson,  1978  &  1979).  Another  field  of  application  of  respiratory 
measures  has  been  the  area  of  vestibular  stimulation.  A  study  by  Bles  et  al.  (1988)  has 
found  high  correlation  between  the  degree  of  motion  sickness,  stress  anxiety  and  end-tidal 
pCOa  decrease  in  subjects  onboard  a  mine-hunter.  It  has  therefore  been  suggested  that 
hyperventilation  be  used  as  an  index  to  study  the  response  to  an  aversive  stimuli. 

In  summary,  the  sensitivity  of  respiratory  measures  is  highly  dependent  on  the  choice  of 
the  parameters.  The  use  of  time  parameters  alone,  like  respiratory  rate,  are  insufficient  to 
detect  changes  with  task  demands.  There  is  a  need  to  use  parameters  dependent  on 
driving  mechanisms,  and  those  influenced  by  chemoreceptor  activity. 

3.7  Musculo-Skeletal  Measurements 

3.7.1  Electromyography 

The  electromyogram  (EMG)  is  a  recording  of  the  action  potentials  generated  when  the 
fibers  of  a  muscle  contract.  The  amplitude  and  frequency  of  the  EMG  signal  in  most 
situations  has  a  positive  linear  relationship  to  the  strength  of  the  muscle  contraction.  The 
most  direct  means  of  communication  between  the  “inner”  and  “outer”  environment  of  an 
individual  can  be  thought  of  as  being  done  by  the  efferent  motor  system.  Muscle  tensions 
occur  in  response  to  physical  motor  demands,  and  are  also  intimately  linked  to  emotional 
and  cognitive  activity.  The  recording  of  muscle  activity  is  therefore  relevant  in  many 
situations,  the  most  important  related  to  the  current  discussion  being  anxiety  and  stress 
response. 

3.7.1 .1  Techniques  for  EMG  Recording  and  Analysis 

There  are  two  fundamental  ways  of  recording  EMG  signals:  by  the  use  of  indwelling 
electrodes  within  the  muscles,  or  through  surface  electromyography,  with  electrodes 
placed  on  the  skin.  The  latter  technique  is  more  commonly  used,  and  can  give  an 
accurate  picture  of  muscle  activity  of  superficial  muscles.  Electrode  placement  is 
determined  through  an  anatomical  analysis  of  die  muscle  of  interest.  The  best  placement 
site  of  an  electrode  is  over  the  muscle  belly,  near  the  motor  point. 

Pairs  of  silver/silver  chloride  electrodes  are  commonly  used,  about  4  to  7  mm  in  diameter 
and  spaced  at  a  distance  of  10-20  mm  apart.  The  skin  is  carefully  abraded  and  prepared 
using  alcohol  and  fine  sand  paper.  The  active  surface  electrodes  are  placed  longitudinally 
over  the  muscle  belly,  with  their  axes  parallel  to  the  muscle  fibers.  Electrode  resistance  is 
recommended  to  be  below  5-7  k  Ohm.  A  third  grounding  electrode  is  placed  over  any 
bony  prominence.  Figure  28  shows  examples  of  electrode  placements  for  the  trapezium 
and  the  forearm  flexor  muscle. 
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Standard  trapezius  lead  Standard  forearm  flexor  lead 


Figure  28:  Typical  Electrode  Locations  for  EMG  Recording  from  the  Trapezium  and  a 

Forearm  Flexor 

The  EMG  signals  are  amplified  using  a  multi-channel  system  and  band-passed  filtered 
with  the  upper  filter  frequency  set  to  about  1  kHz  and  the  lower  to  10  Hz.  The  EMG 
signal  ranges  in  amplitude  between  100  to  1000  pV,  but  may  be  as  low  as  between  2-4 
pV  under  conditions  of  deep  sleep. 

Common  techniques  for  quantifying  an  EMG  signal  are  rectifying  and  signal  integration. 
After  signal  amplification,  the  signals  are  full-wave  rectified,  and  ftien  integrated  with  a 
time  constant  ranging  between  0.5  to  2  seconds.  A  variant  of  signal  integration  is 
criterion  integration.  In  this  procedure,  the  integrated  voltage  is  compared  to  a  preset 
voltage  level  called  the  “criterion”.  Whenever  the  integrated  voltage  reaches  the  criterion 
level,  the  computer  marks  a  spike  on  the  screen.  The  number  of  “marked”  spikes  from 
the  recorded  data  can  help  estimate  the  EMG  activity  during  the  recording  session. 
Examples  of  raw  and  processed  EMG  data  are  shown  in  Figure  29. 


□  . 


Figure  29:  Raw  (A),  Criterion-Integrated  (B),  and  Regular  Integrated  (C)  EMG  Signal 


52 


CYBERNET  SYSTEMS  CORPORATION 


The  power  frequency  spectrum  of  EMG  signals  is  also  frequently  used  to  quantify  the 
muscular  events  occurring  with  fatigue.  The  fast  Fourier  transformation  (FFT)  is  used  to 
convert  the  recorded  time  domain  data  into  the  frequency  based  data.  Each  time  domain 
is  divided  into  10-20  segments,  and  FFT  is  done  for  each  of  the  individual  segments. 
From  this,  the  additional  parameter  of  EMG,  namely  the  mean  power  frequency,  is 
obtained. 


3.7.1 .2  Use  of  EMG  in  Human  Performance/State  Identification 

It  is  now  well  understood  that  EMG  amplitude  is  a  consistent  and  sensitive  measure  of 
motor  unit  recruitment.  Many  studies  have  demonstrated  the  relationship  between  the 
force  generated  by  a  muscle  and  the  mean  signal  recorded  during  isometric  exercise 
(Lippold,  1952)  or  with  contractions  with  constant  velocity  (Bigland  and  Lippold,  1954). 
Linear  relations  between  the  contraction  strength  and  the  integrated  EMG  signal  have  also 
been  established.  Many  analysis  models  have  been  proposed  that  have  an  ability  to 
predict  the  potential  muscle  capability  of  an  individual,  and  thus  help  in  optimal 
placement  of  individuals  in  suitable  jobs.  The  frequency  components  of  EMG  have  been 
observed  to  decrease  with  fatigue  (Piper,  1912),  and  the  power  frequency  spectrum 
analysis  has  been  used  as  an  accurate  method  to  detect  muscular  fatigue. 

Psychophysiological  studies  of  muscle  activity  have  a  long  history.  Involuntary  muscle 
tension  has  been  observed  following  stimulus  presentation  in  a  two-stimulus  reaction 
time  paradigm  (Jennings  et  al.,  1970).  EMG  measures  have  also  been  used  frequently  in 
studies  of  motor  preparation,  which  requires  subjects  to  wait  before  responding. 

Peripheral  measures  such  as  the  EMG  have  been  used  by  Brunia  (1993)  in  comparison 
with  central  measures  like  event  related  potentials.  It  has  also  been  demonstrated  by 
McGuigan  (1978)  that  there  are  specific  changes  in  the  musculature  of  the  speech 
muscles  and  the  writing-arm  muscles  during  “silent”  language  processing,  both  writing 
and  verbalization. 

One  significant  work  on  the  relationship  of  EMG  to  mentation,  is  the  study  by  Caccioppo 
and  Petty  (1981).  They  observed  that  mental  processes  are  accompanied  by  muscle 
activity  in  focal  sites,  especially  at  sites  in  the  musculature  required  for  “acting  out”  one’s 
thoughts.  They  maintain  that  the  amplitude  of  somatic  responses  observed  through  EMG, 
decreases  as  the  distance  of  measurements  from  these  focal  sites  increases. 

Measurement  of  muscle  activity  can  provide  additional  information  on  subject  readiness 
or  stress,  as  well  as  the  levels  of  fatigue.  Some  of  the  commonly  used  sites  have  been  the 
frontalis,  temporalis,  masseter,  and  the  trapeziiis  muscles.  Monitoring  of  muscle  activity 
at  these  sites,  and  the  special  data  processing  techniques  that  are  currently  available,  can 
provide  a  real  time  indicator  of  the  stress  and  fatigue  level  in  the  human  operator. 
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3.7.2  Actimetry  /  Motion  Measurement 

Though  motor  activity  may  not  be  considered  a  psychophysiological  measure,  activity 
monitoring  is  an  essential  since  it  can  affect  physiological  measures.  Actimetry  has  been 
particularly  used  in  studies  on  vigilance,  work  shifts,  and  jet  lag.  The  two  common 
techniques  in  actimetry  are  ambulatory  actimetry  using  a  wrist  actimeter,  and  video 
actimetry  using  video  recordings.  The  former  measures  motor  activity  when  the  subject 
is  free  to  move  around  in  a  physically  large  area,  while  the  latter  is  the  preferred 
technique  when  subject  behavior  and  activity  is  restricted  to  a  confined  setting. 

The  recording  system  consists  of  a  watch-like  unit  that  can  be  worn  around  the  wrist,  leg, 
or  the  body  after  suitable  adjustments  as  shown  in  Figure  30.  A  three  axis  accelerometer 
sensor,  a  preamplifier  and  a  filter  are  incorporated  into  this  system,  which  is  interfaced  to 
a  microprocessor.  There  is  provision  for  standard  analyses  through  software  that  allows 
sleep  state  scoring,  basic  statistics  and  several  forms  of  data  display. 


Wrist  actigraph 


Figure  30:  Ambulatory  Monitoring  Actigraph 

The  unit  can  be  programmed  for  an  epoch  interval,  the  start  delay,  an  event  marker, 
sensitivity  parameters,  and  filter  cutoff  frequencies,  as  well  as  detection  thresholds  and 
gain.  The  human  operator  wears  the  unit  during  the  complete  experimental  period. 
Movements  of  the  wrist  are  recorded  whenever  the  energy  exceeds  a  programmed 
threshold.  There  is  also  provision  for  the  subject  to  record  the  times  at  which  interesting 
events  occur,  by  pressing  a  button. 

Other  techniques  in  actigraphy  have  used  sensors  made  of  strain  gages  that  are  fixed  to 
the  skin  using  adhesive.  Video  display  sensors  along  with  video  cameras  have  also  been 
used  as  recording  devices,  with  CCD  cameras  for  night-time  recording. 
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3.7.2.1  Use  of  Actimetry  in  Human  Performance/State  Identification 

Actimetry  recordings  have  been  used  in  a  study  of  a  French  air  force  crew  during  an 
exercise  operation  for  readiness  (  Le  Menn  et  al.  1988)  in  a  confined  environment  over  a 
period  often  days.  This  was  done  to  assess  the  response  capability  of  the  crew  at  the  end 
of  the  test  period  and  to  test  the  efficiency  of  the  shelters.  Results  from  this  study 
indicated  that  subjects  were  more  active  during  the  day  than  the  night,  and  that  the 
greatest  activity  was  observed  during  the  first  and  the  last  three  days  than  the  middle  four 
days. 

Actigraphy  can  provide  an  additional  perspective  to  the  changes  in  psychophysiological 
parameters  which  may  be  related  to  the  changing  aspects  of  the  activity  fiiat  the  subject  is 
involved  in.  Patterson  et  al.  (1993)  conducted  a  study  to  evaluate  the  reliability  and 
validity  of  the  wrist  actigraph,  where  subjects  performed  various  sedentary  and  physical 
activities.  They  found  that  the  actigraph  reliably  differentiated  between  the  various  types 
of  activities,  both  physical  and  sedentary.  They  also  found  that  the  oxygen  update  and  the 
heart  rate  were  closely  correlated  with  the  actimeter  data. 

3.8  Endocrine  Measurements 

As  mentioned  before,  many  endocrine  systems  respond  to  stress  and  their  fimction  is 
closely  related  to  the  autonomic  nervous  system  activity.  It  is  also  known  that  a  subject’s 
level  of  control  over  a  situation  influences  the  stress  reactions  manifested.  From  an 
application  view  point,  a  subject  who  is  able  to  regulate  stimulus  input  is  better  suited  for 
maintaining  psychological  atousal  and  involvement  at  an  optimal  level  in  a  variety  of 
situations  than  one  without  this  ability.  Studies  have  shown  that  the  plasma  level 
concentrations  of  certain  hormones  are  influenced  by  operationally  stressful 
environments.  The  plasma  level  at  an  initial  phase  of  a  task  situation  may  vary  from  that 
at  a  later  stage,  when  coping  strategies  have  been  established  (Frankenhauser,  1978). 

3.8.1  Hormone  Performance  Markers 

The  three  important  hormonal  indicators  of  stress  are  cortisol,  prolactin  and  testosterone, 
all  of  which  are  sensitive  to  stress  and  high  workload.  Noninvasive  techniques  are 
available  for  measuring  hormonal  level  using  saliva  and  urine.  It  is,  however,  important 
to  minimize  stress  due  to  the  sample  collection  process  by  debriefing  subjects  before  the 
testing,  and  by  using  the  least  invasive  method  of  collection. 

Cortisol  is  a  steroid  hormone  that  has  major  effects  on  carbohydrate  metabolism,  salt  and 
water  balance  and  the  inflammation  response.  Its  level  in  blood  can  be  regarded  as  the 
anticipatory  stress  experienced  by  a  subject.  Johansson  et  al.  (1990)  have  reported 
changes  in  prolactin  level  prior  to  and  following  psychological  stress  in  both  men  and 
women.  Prolactin  has  been  found  to  be  particularly  sensitive  to  stress  anticipation  and 
anxiety.  Testosterone  is  a  sex  hormone  produced  by  the  testes,  ovaries,  and  the  adrenal 
cortex.  Stressfiil  conditions  have  a  suppressive  effect  on  testosterone  secretion. 
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Salivary  analysis  has  been  proven  usefiil  to  determine  the  level  of  cortisol  and 
testosterone.  Secretion  from  salivary  glands  varies  in  composition  from  tiine  to  time, 
especially  as  a  function  of  experimental  condition  and  the  intensity  of  individual 
stimulation.  The  resting  or  the  baseline  secretion  is  the  continuous  secretion  that  takes 
place  in  the  absence  of  obvious  stimulation.  Saliva  can  be  collected  using  a  dental  roll 
placed  in  the  mouth  after  thorough  rinsing.  After  the  dental  roll  is  saturated,  it  is  removed 
and  centrifuged  to  obtain  the  saliva  for  analysis  by  radio-immunoassay.  A  disadvantage 
of  the  dental  roll  procedure  is  that  the  cotton  interferes  with  the  identification  of  several 
hormones,  particularly  testosterone.  The  Oral-Diffusion-Sink  (ODS)  device  is  the  other 
alternative,  which  can  be  used  for  the  in  situ  collection  of  an  ultrafiltrate  of  saliva  and  is  a 
variation  on  an  earlier  design  developed  for  time-integrated  measurement  of 
corticosteroids  in  interstitial  fluid.  It  is  worn  in  the  mouth  and  continuously  accumulates 
the  compounds  of  interest  as  they  diffuse  into  the  device  along  a  concentration  gradient. 
This  method  can  be  used  for  testing  for  cortisol. 

Additional  information  that  needs  to  be  recorded  during  the  collection  of  saliva  is  Ihe 
time  of  day  and  date  of  collection,  physiological  and  psychological  state  of  the  subject, 
and  the  circumstances  under  which  the  sampling  is  done.  There  are  several  commercially 
available  kits  for  radio-immunoassay  of  cortisol,  prolactin  and  testosterone.  Analysis  of 
the  levels  of  these  hormones  is  one  way  of  determining  the  impact  of  stressors  on  the 
subjects,  as  well  as  the  influences  of  the  environment. 

Considerable  insight  into  the  adaptive,  psychological  and  biological  reactions  that  allow 
subjects  to  cope  with  work-related,  environmental,  and  other  stressors  can  be  obtained 
using  hormonal  analysis.  Biological  fluid  analysis  however  does  require  assistance  from 
individuals  with  significant  biochemical  training  since  it  is  easy  to  obtain  incorrect  results 
due  to  use  of  faulty  procedures. 

3.9  Subjective  Evaluation 

There  are  two  general  approaches  that  can  be  applied  to  the  assessment  of 
cognitive/mental  workload:  objective  measures  and  subjective  evaluation.  Objective 
measures  are  based  purely  on  measurable  parameters  and  experimental  methods,  this 
includes  the  array  of  psycho-physiological  measurements  previously  discussed.  The 
subjective  approach,  on  the  other  hand,  gathers  data  from  the  subject  about  the  task 
experience  through  ratings,  rankings,  questionnaires  etc. 

Subjective  evaluation  methods  are  mostly  used  by  practicing  ergonomists,  and  involve 
self-reporting  by  the  task  performers  or  product  users,  Examples  include:  ranking  and 
rating  methods,  questionnaires,  interviews  and  checklists  which  involve  asking  the 
subject  how  he/she  feels  about  the  workload  level  of  the  task.  Recently,  a  growing  interest 
in  the  nature  and  capabilities  of  human  performance  has  begun  to  suggest  that 
supplementing  objective  measures  with  subjective  evaluation  can  provide  a  better 
evaluation  of  human  performance  (Jex,  1988). 
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Subjective  rating  is  actually  one  of  the  easiest  ways  to  estimate  the  mental  workload  of  a 
person.  The  subjective  assessment  involves  asking  the  subject  what  he  or  she  feels  about 
the  load  of  the  task.  The  subject  is  usually  given  instructions  for  how  to  rate  loads,  as  well 
as  definitions  of  the  various  rating  scales,  and  an  example  rating  scale  and/or 
interview/questionnaire.  The  subject  then  rates  the  load  with  reference  to  predetermined 
levels  (i.e.  high/low,  good/poor,  no  effort/enormous  effort)  determined  by  the  researchers. 

Figure  31,  below,  lists  several  subjective  evaluation  methods  that  can  be  used  to  assess 
operator  workload  in  various  human-machine  environments. 
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Figure  31:  Subjective  Evaluation  Methods 


Method 

Technique 

Outcome  Measures 

NASA  Bipolar* 

Rating 

Overall  workload, 
performance,  frustration, 
task  difficulty,  fatigue 

Multidimensional  bipolar 
rating  scale^ 

Rating 

Overall  workload, 
frustration  level,  stress, 
performance,  fatigue _ 

NASA-TLX^ 

Rating 

Overall  index  of  mental 
demand,  physical  demand, 
temporal  demand, 
performance,  effort, 
frustration 

SWA'T’'’ 

Rating 

time  load,  mental  load, 
stress  load 

MCH*” 

Rating 

task  difficulty 

McDormell  Rating  Scale' 

Rating 

control  difficulty, 
attentional  demand _ 

Bedford  Workload  Scale 

Rating 

overall  workload,  task 
difficulty, 
perceived  exertion 

Borg  Scale** 

Rating 

perceived  exertion 

*  NASA-TLX=  NASA-Task  Load  Index;  SWAT=  Subjective  Workload  Assessment 
Technique;  MCH=  Modified  Cooper-Harper. 


'  American  National  Standard.  (1992).  Guide  to  Human  Performance  Measurements.  BSR/AIAA,  G-035- 

2  Hamlr  JR  Childress,  M.E.,  and  Hart,  S.G.  (1982).  Rating  consistency  and  component  salience  in 
fubjeciive  workload  estimation.  Paper  presented  at  the  18*  Annual  Conference  on  Manual  Control, 

^  S.a  Id  Staveland,  L.E.  (1988).  Development  of  a  NASA-TLX  (Task  Load  Index):  results  of 

empirical  and  theoretical  research.  In  Human  Mental  Workload,  edited  by  P.A.  Hancock  and  M. 

^id^B^  and  Nygren.  T.E.  (1988).  The  subjective  workload  assessment  technique:  ^ 
^fm^^rZ^eh  workload.  In  Human  Mental  Workload,  edited  by  P.A.  Hancock  and  M.  Meshkat, 

C.A.,  Nygren,  T.E.  and  Eggemeier  F.T.  Dev^ent  of 

htidimensional  subjective  measures  of  workload.  Proc.  of  the  Human  Factors  Society,  p.  403-406 
‘  Wierwille,  W.W.  aid  Casali,  J.G.  (1983).  A  validated  rating  scale  for  global  mental  workload 

anulications.  Proc.  Of  the  Hmnan  Factors  Society,  pp.  129-133  ^  r  tj  jv 

’’  McDonnell,  J.D.  (1968).  Pilot  Rating  Techniques  for  the  Estimation  and  Evaluation  of  Han  mg 

*  BoT^^G^A.^tf^^Psychological  bases  of  perceived  exertion.  Medicine  and  Science  in  Sports  and 
Exercise,  4,  377-381 
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The  NASA  bipolar  and  multidimensional  bipolar  rating  scales  can  be  used  to  examine 
differences  in  individual  subjective  responses  and  associations  between  subjective  and 
objective  measures.  The  multidimensional  scale  utilizes  several  bipolar  scales:  overall 
workload,  frustration  level,  stress,  performance  and  fatigue. 

The  NASA-TLX  is  one  of  the  most  widely  used  methods  for  assessing  mental  workload. 
Developed  by  NASA,  this  technique  measures  six  workload-related  factors  scales:  mental 
demand,  physical  demand,  temporal  demand,  performance,  effort  and  frustration.  What 
can  be  determined  from  the  NASA-TLX  scale  are  the  following:  differences  in  subjective 
and  objective  responses  and  their  respective  magnitudes  and  individual  information  about 
the  task.  Figure  32  and  Figure  33  illustrate  the  NASA-TLX  workload  assessment 
methodology. 

SWAT  was  first  developed  by  the  Armstrong  Medical  Research  Laboratory  (AARML) 
for  the  purpose  of  assessing  pilot  workload.  SWAT  measures  mental  workload  by 
measuring  how  hard  one  has  to  work  on  a  certain  task.  The  three  key  measures  for  SWAT 
are:  time  load,  mental  effort  load,  and  psychological  stress  load. 

MCH  was  first  developed  to  assess  flight  characteristics  and  to  evaluate  the  “flyability”  of 
aircraft.  This  scale,  which  is  a  modified  version  of  the  Cooper-Harper  scale,  is  considered 
a  sequential  rating  process.  The  controllability  and  then  the  performance  are  assessed. 

Any  modifications  needed  with  respect  to  pilot  workload  optimization  are  made  to 
achieve  the  desired  performance. 

The  McDonnell  rating  scale,  similar  to  the  Cooper-Harper  scale,  rates  aircraft  handling 
properties.  Ratings  are  “linearized”  so  that  statistical  techniques  can  be  applied  to  all 
rankings. 

The  Bedford  rating  scale  and  the  Borg  scales  measure  perceived  exertion.  These  two 
scales  give  a  cross  comparison  to  the  physical  measures  and  allow  the  investigator  to 
explore  how  the  job  feels  to  the  employee  -  or,  in  other,  words  the  relationship  between 

the  physical  stimuli  and  the  perception  of  their  intensity. 
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RATING  SCALE  DEFINITIONS 

Title 

Endpoints 

Descriptions 

MENTAL  DEMAND 

Low/High 

How  much  mental  and  perceptual  activity 
was  required  (e.g.,  thinking,  deciding, 
calculating,  remembering,  looking, 
searching,  etc.)?  Was  the  task  easy  or 
demanding,  simple  or  complex,  exacting 
or  forgiving? 

PHYSICAL 

DEMAND 

Low/High 

How  much  physical  activity  was  required 
(e.g.,  pushing,  pulling,  turning,  controlling, 
activating,  etc.)?  Was  the  task  easy  or 
demanding,  slow  or  brisk,  slack  or 
strenuous,  restful  or  laborious? 

TEMPORAL 

DEMAND 

Low/High 

How  much  time  pressure  did  you  feel  due 
to  the  rate  or  pace  at  which  the  tasks  or 
task  elements  occurred?  Was  the  pace 
slow  and  leisurely  or  rapid  and  frantic? 

EFFORT 

Low/High 

How  hard  did  you  have  to  work  (mentally 
and  physically)  to  accomplish  your  level  of 
performance? 

PERFORMANCE 

Good/Poor 

How  successful  do  you  think  you  were  in 
accomplishing  the  goals  of  the  task  set  by 
the  experimenter  (or  yourself)?  How 
satisfied  were  you  with  your  performance 
in  accomplishing  these  goals? 

FRUSTRATION 

LEVEL 

Low/High 

How  insecure,  discouraged,  irritated, 
stressed  and  annoyed  versus  secure, 
gratified,  content,  relaxed  and  complacent 
did  vou  feel  during  the  task? 

Figure  32:  Rating  Scale  Definitions:  NASA-TLX  Method  for  Subjective  Workload 

Assessment 
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MENTAL  DEMAND 

1  1  1  1  1  1  1  iJ 

1  1  1 

1  1  1  1 

J_U 

Low 

ffigh 

PHYSICAL  DEMAND 

1  1  1  1  1  1  1  iJ 

1  1 1 

1 1  1  1 

_LU 

Low 

ffigh 

TEMPORAL  DEMAND 

1  1  1  1  1  1  1  1 

1  1 

1 1  1  1 

_LU 

Low 

ffigh 

PERFORMANCE 

1  1  1  1  1  1  1  1 

1  1 

1  1 1  1  1 

lLU 

Good 

Poor 

EFFORT 

1  1  1  1  1  1  1  .l_ 

1  1  1 

Mill 

1  1  1  i 

Low 

ffigh 

FRUSTRATION 

1  1  1 1  1 1  ij 

1  1  1 

Mil 

1  1  1  J 

Low 

ffigh 

Figure  33:  Sample  Rating  Sheet:  NASA-TLX  Method  for  Subjective  Workload  Assessment 
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3.10  Processing  and  Analysis  of  Measured  Data 

The  nature  of  the  psycho-physiological  parameters  that  can  be  measured  were  discussed 
in  detail  in  the  previous  sections.  Here  we  present  a  brief  review  of  the  types  of  analyses 
that  can  be  performed  on  these  data  in  order  to  derive  indicators  human  performance. 

Electrocardiogram  (ECG) 

Rate  detection  algorithms  can  be  applied  to  the  ECG  in  order  to  determine  heart  rate 
parameters.  Basic  Heart  Rate  (HR)  is  computed  as  beats  per  minute,  and  is  generally 
averaged  over  a  period  of  time  (i.e.  1  minute).  Inter-Beat  Interval  (IBI)  is  a  measure  of 
instantaneous  heart  rate  that  is  computed  at  each  beat.  Variations  in  HR/IBI  can  then  be 
examined  for  patterns  indicative  of  stress  level  and  workload.  An  advanced  method  for 
examining  heart  rate  variability  is  the  Heart  Rate  Power  Spectral  Analysis  (HRPSA), 
which  provides  a  frequency  domain  view  of  the  variability. 

Electromyogram  (EMG) 

EMG  is  generally  processed  by  performing  a  power  calculation  (sometimes  within  a 
specific  frequency  band)  over  successive  windows/epochs  to  provide  a  measure  of  muscle 
tension.  Values  may  be  normalized  against  full  scale  power  (i.e.  when  muscle  tension  is 
at  maximum)  in  order  to  provide  a  more  quantitative  assessment.  Muscle  tension 
variations  can  dien  be  examined  for  patterns  indicative  of  stress  level  and  workload. 

Respiration 

A  rate  detection  algorithm  can  be  applied  to  the  respiration  signal  in  order  to  determine 
respiration  rate.  In  addition,  other  respiratory  parameters  may  also  be  estimated  from  the 
respiration  signal,  including:  tidal  volume,  inspiration  time,  expiration  time,  and 
expiratory  pause  time.  It  may  prove  useful  to  combine  these  respiration  parameters  into  a 
single  measure  of  respiration  regularity  or  other  indicator.  Variations  in  respiration  rate 
and  other  respiration  measures  can  then  be  examined  for  patterns  indicative  of  stress  level 
and  workload. 

Galvanic  Skin  Response  (GSR) 

The  GSR  is  a  direct  measure  of  skin  impedance  and  needs  no  specific  processing.  Skin 
impedance  variations  can  be  directly  examined  for  patterns  indicative  of  stress  level  and 
workload. 

Electro-oculogram  (EOG) 

The  EOG  is  generally  processed  to  determine  absolute  eye  position,  specific  eye  motion 
characteristics  (such  as  REM  activity),  and  blink  detection.  Eye  position  can  be 
determined  from  calibrated  levels  of  the  absolute  DC  voltage  value,  and  may  be  used  to 
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confirm/correlate  eye  tracker  measurements  (if  also  performed).  Specific  eye  motions  can 
be  determined  by  examining  power  in  specific  firequency  ranges,  for  instance:  saccades, 
accomodations,  or  REM  activity.  Blink  detection  can  be  determined  by  identification  of 
its  distinctive  waveform  shape  and  then  used  along  with  a  rate  determination  algorithm  to 
produce  blink  rate.  Once  eye  position,  eye  motion,  and/or  blink  rate  are  determined,  they 
can  then  be  examined  for  patterns  indicative  of  stress  level  and  workload. 

Electroencephalogram  (EEG) 

The  EEG  is  a  highly  complex  signal  and  has  an  array  of  possible  analysis  techniques 
(both  well  established  and  those  under  research).  A  standard  method  for  EEG  signal 
analysis  is  examination  of  power  level  in  the  traditional  frequency  bands:  alpha  (8-12 
Hz),  beta  (13-30  Hz),  theta  (4-7  Hz),  and  delta  (<4  Hz).  These  specific  fi-equency  bands 
have  historically  been  used  because  activity  in  tihese  bands  has  been  shown  to  occur 
during  different  mental  states.  Examination  of  power  levels  in  each  of  these  bands  (and/or 
shifts  of  power  between  bands)  is  one  method  that  can  be  examined  for  indicators  of 
cognitive  workload. 

Much  research  and  development  has  been  performed  towards  the  development  of 
methodologies  and  algorithms  to  classify  mental/cognitive  states  using  the  EEG  signal. 
These  methods  may  include  utilization  of  the  traditional  frequency  band  powers,  or  more 
advanced  information  such  as  the  full  frequency  spectrum  or  time  series  based  parameters 
of  the  signal.  Topographic  features  of  the  EEG  (i.e.  spatial  distribution  of  EEG  activity 
over  different  regions)  are  also  important  for  analysis  and  classification  of  the  EEG. 
Several  methods  for  spatial  analysis  of  EEG  were  presented  in  Section  3.2.10. 

In  addition  advanced  methods  for  EEG  analysis  that  are  still  primarily  undergoing 
continued  research  (rather  than  widespread  clinical  use)  include  time-frequency, 
bispectral,  and  coherence  analyses. 

Evoked  Potentials 

Evoked  potentials  are  EEG  responses  elicited  by  a  specific  stimulus  or  event,  and  are 
buried  in  other  ongoing  EEG  activity  not  associated  with  the  stimulus/event.  Since  the 
evoked  potential  has  a  low  magnitude  relative  to  the  other  ongoing  EEG  activity  it  must 
be  extracted  from  this  background  EEG  “noise”.  This  is  typically  performed  by  averaging 
multiple  responses  to  the  same  stimulus/event  in  order  to  improve  the  signal  to  noise  ratio 
of  the  evoked  potential.  Once  an  evoked  potential  with  sufficient  signal  to  noise  ratio  is 
obtained,  it  is  then  usually  characterized  by  amplitude  and  latency  of  significant  peaks  in 
the  response.  These  peaks  have  been  shown  to  change  due  to  different  cognitive  states 
and/or  disorders.  Other  advanced  methods  that  may  be  considered  include  spectral 
analysis,  time-frequency  analysis,  and  bispectral  analysis  of  the  evoked  potential  signal. 
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4.  Recap  of  Phase  I  Effort  and  Results 

This  section  provides  a  brief  review  of  the  Phase  I  effort,  the  foundation  it  provided  for 
continued  work,  and  how  this  was  transitioned  into  the  Phase  II  effort. 

The  basic  goal  of  the  Phase  I  effort  was  to  address  the  need  for  a  systematic  approach  and 
comprehensive  methodology  for  more  effective  design  of  human-operator  interfaces. 

More  specifically,  we  set  out  to  identify  a  path  of  research  and  development  that  would 
lead  to  the  implementation  of  a  real-time  human  performance  measurement  system.  This 
human  performance  measurement  system  would  provide  valuable  insight  into  the  human 
operator’s  performance  level,  and  include  assessments  such  as  attention,  fatigue,  stress, 
cognitive  state,  and  workload. 

We  envisioned  that  this  technology  would  ultimately  be  applied  to  the  task  of 
systematically  designing  and  evaluating  interfaces  for  advanced  military  computer-based 
operations.  As  military  computer  systems  and  dieir  corresponding  interfaces  grow  more 
complex,  the  operator’s  (soldier’s)  task  is  also  becoming  more  difficult.  Interface  designs 
that  take’into  account  the  operator’s  cognitive  processing  will  provide  a  higher  level  of 
cognitive  congruence  between  the  system  the  operator  is  using  and  the  operator  s  existing 
internal  mental  models  of  environment  and  situation. 

This  Phase  I  proposal  identified  following  set  of  Technical  Objectives: 

Develop  “Human  Performance  Measurement”  Technology  Roadmap 

This  research  survey,  or  ‘roadmap’ ,  had  the  following  criteria: 

•  Survey  academic,  commercial,  and  government  research. 

•  Identify  all  relevant  human  performance  related  sensory,  psychomotor, 
physiological  and  cognitive  processes  relevant  to  evaluating  human  performance. 

•  Identify  research  issues,  challenges,  etc.,  associated  with  measuring  these  human 
processes  relative  to  human  performance  evaluation. 

•  Identify  potential  representative  (military  and  private  sector)  technology 
applications  (current  /future)  for  overlay  on  the  roadmap. 

•  Identify  data  collection  hardware  and  software  issues  and  challenges  associated 
with  identified  applications. 

Demonstrate  “Human  Performance  Measurement  System”  Technology 

The  Phase  I  effort  included  a  brassboard  system  which  demonstrated  Cybemet’s 
existing  capabilities  in  each  of  the  critical  technologies.  These  capabilities  became 
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the  Phase  II  building  blocks  upon  which  a  full  scale  performance  measurement  system 
was  ultimately  developed. 

Develop  Preliminary  Phase  II  Plan 

This  objective  entailed  the  development  of  a  prototype  "human  performance 
measurement  system"  design  for  development  and  evaluation  in  the  Phase  11.  Along 
with  this  design  was  a  preliminaiy  plan  for  carrying  out  the  Phase  n  effort. 


The  Phase  I  effort  was  highly  successful  in  achieving  these  objectives,  and  in  fact 
broadened  the  scope  of  the  proposed  concept  to  encompass  a  wider  range  of  human 
operator  performance  applications.  The  Phase  I  effort  accomplished  the  following  tasks: 

.  Surveyed  broad  set  of  human  performance  related  research 

•  Developed  a  human  performance  measurement  taxonomy 

•  Defined  data  collection  hardware/soflware  requirements  and  issues 

•  Identified  wide  range  of  candidate  human  operator  applications 

•  Demonstrated  the  concept  with  a  Phase  I  prototype  system 

•  Designed  a  preliminary  Phase  II  system 

•  Developed  a  Phase  n  implementation  and  validation  plan 

Figure  34  illustrates  the  overall  concept  of  the  total  human  performance  assessment 
concept  that  resulted  from  the  Phase  I  effort. 


Expanded  concept  to  include 
broader  human  performance 
assessment 


•  Cognitive  Measures 

•  Physical  Performance 

•  Ps^hologicai  Factors 

•  Workload 

•  Atiention 

•  Sbess 


Applicable  to 
any  human  task 
environment 


Total  Human 
Performance 
Assessment 


Cognitive 

Measurement 

•Cognitive  Capacity 
•Cognitive  Overload 


Performance 

Measurement 

•Workload 

•Stress 

•Attention 

•Physical 

•Psychological 


Figure  34:  Illustration  of  the  Concept  for  Total  Performance  Assessment  that  Resulted 

from  the  Phase  I  Effort 
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5.  Overview  of  Phase  II  Objectives  and  Accomplishments 

The  Phase  n  effort  set  out  to  accomplish  the  following  ultimate  goal; 

Develop  and  implement  a  highly  flexible,  modular  system  for  human  performance 
measurement,  capable  of  evaluating  human  performance  throughout  a  broad 
range  of  application  environments 

The  approach  taken  was  to  integrate  an  array  of  state-of-the-art  measurement  systems  and 
data  Collection/analysis  software  into  a  comprehensive  system  for  performance 
assessment.  This  successful  development  built  off  of  several  prior  and  ongoing  Cybernet 
R&D  and  product  development  efforts  for  the  following  subsystems: 

Physiological  measurement  systems 
Eye  tracking  systems 
Motion  capture  system 

Distributed  data  collection  and  analysis  software 
Other  data  capture  devices 

Through  the  development  and  integration  of  this  system  we  incorporated  methods  for 
both  measurement  and  analysis  of  the  identified  array  of  performance-related  parameters 
(physiological,  psychological,  and  performance  measures).  The  Phase  n  effort  also 
addressed  validation  and  application  testing  by  conducting  an  initial  set  of  experiments 
aimed  at  identification  of  performance-related  indicators. 

The  Phase  II  proposal  identified  the  following  set  of  broad  objectives: 

1.  Requirements  definition 

•  Review  the  Phase  I  study  and  design  specification 
.  Re-evaluate  the  latest  state-of-the-art  technology 

.  Prepared  an  updated  design  specification 

2.  Development  of  the  complete  Human  Performance-Based  Measurement 
System  (HPBMS) 

.  Develop  a  modular  system  for  application  to  a  wide  range  of  human  operator 
task  environments 

•  Define  the  basic  structure  and  essential  hardware  and  software  components 
.  Develop  specific  measurement  and  analysis  solutions  for  assessment  of  a 

wide  range  of  performance  parameters  (both  physical  and  cognitive) 

3.  Implementation  and  evaluation 

.  Construct,  integrate,  and  test  a  comprehensive  modular  HPBMS  systein 

•  Implement  and  test  HPBMS  scenario  specifically  tailored  for  HCI  applications 
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•  Implement  and  test  HPBMS  scenario  specifically  tailored  for  mobile  task 
applications 

•  Perform  array  of  system  validation  tests 

.  Conduct  preliminary  set  of  research-oriented  performance  assessment 
experiments 

Each  of  these  objectives  was  successfully  accomplished  through  this  Phase  II  effort.  The 
following  details  the  specific  accomplishments  achieved. 


Establishfu  Cvbernet’s  Human  Performance  Laboratory 

At  the  outset  of  fiiis  Phase  n  effort  we  established  a  controlled  “Human  Performance” 
laboratory  at  Cybemet’s  facilities.  This  laboratory  was  used  for  the  progressive 
implementation,  integration,  evaluation,  and  experimental  testing  of  the  developed 
HPBMS. 


Figure  36:  Comprehensive  HPBMS  Implementation  for  HCI  Task  Experiments 
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Integrated  Cybernet’s  16  Channel  Physiological  Measurement  System 

As  a  key  component  of  the  developed  HPBMS,  Cybemet’s  16  channel  portable 
physiological  measurement  system  (PPMS)  was  the  first  components  to  be  integrated. 

The  16  channel  device  is  the  physiological  (and  other  sensor)  data  acquisition  system  for 
use  in  desktop  environments  (i.e.  computer  interface  tasks).  This  system  was  designed  to 
operate  within  the  DCAE  software  system  architecture  for  distributed  data  collection  and 
analysis  operations. 

Through  this  Phase  n  effort,  extensive  software  enhancements  to  the  PPMS  and  the 
associated  DCAE  software  components  were  performed  in  order  to  meet  the  specific 
needs  of  the  HPBMS.  Specific  details  of  the  PPMS  are  provided  in  Section  6.6. 

Completed  Cybernet’s  8  Channel  Physiological  Measurement  and 
Integrated  into  the  HPBMS 

The  goal  of  the  HPBMS  to  provide  performance  assessment  in  mobile  task  environments 
(such  as  dismounted  soldier  operations)  presented  the  need  for  a  more  wearable  version 
of  the  physiological  measurement  system.  Through  this  Phase  11  effort,  we  helped 
complete  the  implementation  of  an  8  channel  physiological  measurement  system,  which 
is  a  smaller,  lighter,  and  more  power  efficient  version  of  the  16  channel  PPMS. 

The  8  channel  PPMS  was  designed  for  use  in  mobile  task  environments  where  a 
wearable,  untethered  acquisition  system  is  required.  As  with  the  16  channel  device,  this 
system  operates  within  the  DCAE  distributed  collection  and  analysis  architecture,  in  the 
same  manner.  Specific  details  of  the  PPMS  operations  are  provided  in  Section  6.6. 

Integrated  and  Enhanced  Cybernet’s  Eye  Tracking  System 

The  eye  tracking  system  uses  a  video  based  approach,  based  on  a  device  manufactured  by 
ISCAN,  Inc.  (custom  modified  by  Cybernet)^  and  specialized  eye  tracking  algorithms 
developed  by  Cybernet.  Through  this  Phase  II  effort  we  made  significant  improvements 
to  the  image  processing  algorithms  and  greatly  enhanced  the  user  interface  and 
operational  features.  Using  this  software,  eye  tracking  results  can  be  measured,  and 
synchronously  captured  along  with  other  performance  measurements,  through  one  or  both 
of  two  different  methods. 

The  first  method  utilizes  the  VCR  interface  to  record  video  images  of  the  eye  tracking 
results.  These  video  images  provide  a  real-time  display  of  the  tracked  pupil,  and  a  view  of 
the  user’s  field  of  view  with  an  overlaid  cross  hair  indicating  where  the  user  is  looking.  In 
the  second  method,  numerical  results  of  the  eye  tracking  are  captured  within  the 
synchronized  networked  data  collection  structure  along  with  all  other  measured 
parameters.  This  was  accomplished  through  the  development  of  a  custom  software  driver 


’  Cybernet  has  also  developed  our  own  eye  tracking  device  which  will  be  available  for  integration  into  the 
HPBMS  in  the  very  near  future. 
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and  interface  that  acquires  eye  tracking  results  through  the  DCAE  software.  Data 
provided  by  the  eye  tracking  system  can  include:  pupil  location,  gaze  location,  gaze 
location  vdthin  a  tagged  area  (see  description  of  focus  of  attention  algorithms  that 
follows),  pupil  size,  and  blink.  Details  of  the  eye  tracking  system  are  provided  in  Section 

6.7. 

Developed  and  Implemented  Automated  Focus  of  Attention  Algorithms 

The  automated  focus  of  attention  algorithms  were  developed  through  this  Phase  II  effort, 
as  a  key  enhancement  to  the  eye  tracking  system,  in  order  to  meet  the  requirements  of  the 
HPBMS.  Prior  to  development  of  the  focus  of  attention  algorithms  evaluation  of  a  user  s 
point  of  regard  had  to  be  performed  by  a  observer  watching  the  recorded  video  and 
making  note  of  where  the  user  is  looking.  Now,  the  focus  of  attention  dgorithms  provide 
for  quantitative  determination  of  gaze  location  within  a  computer  monitor  (or  other 
“tagged”  region  within  the  user’s  field  of  view). 

Quantitative  determination  of  the  user’s  point  of  regard  is  of  significant  advantage 
because  it  allows  for  automated  analysis  and  numerical  manipulation  of  a  user’s  gaze 
location.  For  instance,  one  can  easily  perform  calculations  of  how  often,  how  many  times, 
or  how  long  a  user  looks  at  a  certain  object  or  area.  This  has  tremendous  advantage  for 
the  performance  assessment  application.  Quantitative  results  of  the  focus  of  attention  are 
output  via  the  DCAE  interface  previously  discussed.  Details  of  the  eye  tracking  system 
and  the  focus  of  attention  algorithms  are  provided  in  Section  6.7. 

Developed  interface  and  software  components  for  motion  capture  system 

Through  separate  research  and  development  efforts  Cybernet  has  developed  an  optical 
motion  tracking  system,  called  the  FireFly,  which  is  now  available  for  commercial 
purchase.  At  the  outset  of  this  Phase  II  effort  it  was  decided  that  the  HPBMS  should 
possess  the  capabilities  to  interface  to  this  device  to  allow  for  the  measurement  of  hum^ 
body  position  and  motion,  as  related  to  performance  assessment.  Therefore,  teough  this 
Phase  II  effort  we  developed  the  necessary  software  components  for  acquisition,  analysis, 
and  display  of  position  data  within  the  HPBMS.  Using  the  FireFly  system,  the  HPBMS 
can  acquire  time-stamped,  multi-tag  (up  to  32)  position  capture  within  the  synchronized 
DCAE  architecture.  Details  of  the  motion  tracking  system  and  the  developed  interface  are 
presented  in  Section  6.8. 

Developed  and  Enhanced  Data  Collection  and  Analysis  Environment 
Software 

Through  this  Phase  II  effort  we  conducted  an  extensive  software  development  effort  for 
enhancement  of  the  pre-existing  DCAE  software  foundation  and  for  creation  of  numerous 
new  features  and  components  within  the  DCAE. 
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The  DCAE  software  is  a  highly  complex  system  for  real-time,  networked  data  collection 
and  analysis  that  must  balance  competing  demands  for  computational  resources  across  a 
highly  distributed  network.  Updates  to  the  fundamental  software  architecture  and  existing 
foundation  focused  primarily  on  speed,  performance,  and  reliability  enhancements.  The 
results  of  these  enhancements  yielded  greatly  improved  performance,  such  as  increased 
bandwidth  capabilities,  reduced  delay  times,  etc.  Furthermore,  we  made  extensive 
improvements  to  the  user  interfaces  that  have  resulted  in  a  highly  complex  system  that  is 
surprisingly  user  friendly,  easy  to  understand,  and  straightforward  to  operate.  . 

In  addition,  many  new  features  were  added  to  meet  the  specific  requirements  of  the 
HPBMS  application.  Specifically,  we  developed  an  array  of  new  system  interface  drivers 
for  acquisition  of  performance  related  data.  These  include  the  eye  tracking  system 
interface,  the  body  tracking  system  interface,  the  updated  VCR  control  interface,  and  the 
updated  physiological  measurement  system  interface.  An  array  of  signal  analysis 
fhnctions  were  also  developed  to  meet  the  processing  needs  of  the  human  performance 
related  data.  These  include  fimctions  such  as  heart  rate  determination  (from  a  measured 
EKG),  respiration  rate  determination,  artifact  detection,  filtering  and  smoothing 
operations,  etc. 

Other  new  components  developed  for  the  DCAE  software  system  include  specialized  data 
and  signal  viewing  tools.  Specific  viewers  required  by  the  HPBMS  that  were  developed 
include:  an  EEG  topographical  brain  map  viewer,  a  evoked  potential  viewer,  a  body 
tracker  data  viewer,  and  a  tag  editor/viewer.  Enhancements  were  also  made  to  previously 
existing  viewer,  such  as  the  multi-channel  strip  chart  viewer,  in  order  to  meet  the 
requirements  of  the  HPBMS. 

Specific  details  of  the  DCAE  software  system  are  addressed  in  Section  6.5. 

Developed  and  VAi-inATF.n  Evoked  Potential  Measurement  Scheme 

Given  particular  interest  in  the  cognitive  assessment  capabilities  of  the  HPBMS,  the 
ability  to  acquire,  process,  and  evaluate  Evoked  Potentials  (EPs)  was  addressed  through 
this  Phase  11  effort.  Specifically,  we  developed  and  implemented  a  protocol  for  EP 
measurement  utilizing  the  physiological  measurement  system  and  custom  developed 
DCAE  components  for  processing  and  display  of  the  averaged  EP  waveform. 

Acquisition  of  the  stimulus  trigger  signal  is  performed  using  one  of  the  channels  of  the 
PPMS  (while  EEG  is  recorded  on  other  channels),  and  a  DCAE  signal  viewer  was 
developed  for  the  processing  and  display.  The  viewer  provides  for  display  of  successive 
averaged  EP  waveforms,  each  labeled  wifii  the  amplitude  and  latency  of  significant  peaks, 
as  is  customary  for  EP  signal  evaluation.  Functionality  of  the  EP  measurement  protocol 
was  validated  using  a  visual  evoked  potential  presented  on  a  computer  monitor. 
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Conducted  Driving  Simulator  Experiments 

As  a  preliminary  evaluation  and  assessment  of  the  HPBMS  concept,  we  conducted 
driving  simulator  (computer  interface)  experiments,  during  the  early  stages  of  the  Phase  II 
system  integration  process.  This  effort  was  performed  in  conjxmction  with  an  Air  Force 
SBIR  project  that  was  focused  on  the  development  of  innovative  cognitive  capacity 
measurement  system.  Measurements  performed  were:  multi-channel  EEG  (brain  activity), 
ECG  (cardiac  activity),  respiration,  EMG  (muscle  activity),  and  EOG  (eye  motion).  This 
preliminary  evaluation  demonstrated  the  feasibility  and  potential  benefit  of  EEG  and 
other  measures  for  cognitive  and  other  performance  assessment.  Results  are  presented  in 
Section  7.1. 

Conducted  HMMWV  Driving  Experiments 

As  another  preliminary  evaluation  and  assessment  of  the  HPBMS  concept,  we  conducted 
experiments  of  a  human  driving  task  using  a  HMMWV  on  a  standard  test  track.  This 
experiment  was  conducted  jointly  with  another  ARL  SBIR  project  which  was  focused  on 
the  development  of  a  model  for  human  driving.  This  experiment  evaluated  the 
measurement  of  EOG,  ECG,  EMG,  and  within  this  type  of  performance  assessment 
environment.  In  addition  to  providing  indications  of  potential  benefit,  the  results  of  this 
experiment  provided  initial  feedback  on  the  operational,  procedural,  and  functional  issues 
involved  in  the  execution  of  performance  assessment  experiments  in  the  field,.  This 
helped  guide  continued  development  and  enhancement  of  the  HPBMS.  Results  of  these 
experiments  are  presented  in  Section  7.2. 

Conducted  Computer  interface  task  experiments 

This  set  of  experiments  was  conducted  toward  the  latter  part  of  the  Phase  II  effort,  once 
the  comprehensive  HPBMS  had  been  fully  integrated  and  validated.  These  experiments 
therefore  exercised  the  complete  array  of  standard  HPBMS  tools.  A  protocol  was 
designed  to  analyze  human  performance  in  standard  computer  interface  tasks,  to  achieve 
both  a  demonstration  of  the  HPBMS  capabilities  and  to  conduct  a  preliminary 
investigation  into  potential  indicators  of  human  performance. 

The  result  of  this  experimental  protocol  was  highly  successful.  It  demonstrated  the  power 
of  die  developed  HPBMS  as  a  research  tool,  and  identified  specific  areas  for  continued 
research.  Details  of  these  experiments  and  the  results  are  presented  in  Section  7. 

vAi.TnATF.D  Mobile  Environment  Operation 

To  validate  operation  of  the  HPBMS  for  evaluation  of  mobile  tasks,  we  examined 
functionality  of  the  physiological  measurement  system  as  a  wearable  data  collection 
system.  Testing  performed  on  the  8  channel  physiological  measurement  system  device 
(which  is  intended  for  the  wearable  application)  included  a  basic  shake  test  and  an 
examination  of  the  communications  range  using  the  currently  configured  ivireless 
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networking  card.  We  also  determined  the  capabilities  zind  limitations  of  currently 
available  laptop  systems  for  real-time  viewing  of  measured  data  over  the  limited 
bandwidth  wireless  network. 

Conducted  System  Training  Session  for  ARL  Representatives 

A  comprehensive  training  procedure  was  conducted  towards  the  end  of  the  Phase  II  effort 
to  demonstrate  and  teach  operation  of  the  HPBMS  to  ARL  representatives.  The  initial 
training  session  was  held  at  Cybemet’s  facilities,  and  covered  complete  use  of  all 
HPBMS  subsystems  and  operations.  Feedback  was  received  from  the  ARL 
representatives  and  used  to  make  a  few  enhancements  to  the  software  prior  to  delivery. 

Documentation 

As  part  of  this  Phase  11  effort  extensive  documentation  was  prepared,  including  user’s 
manuals  for  the  HPBMS  as  a  whole,  for  each  subsystem,  and  for  all  components  of  the 
data  collection  and  analysis  software.  Troubleshooting  guides  and  other  application  notes 
were  also  developed. 
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6.  Detailed  Description  of  the  HPBMS  System 

6.1  HPBMS  Overview 

The  Human  Performance  Based  Measurement  System  (HPBMS)  is  a  comprehensive  set 
of  hardware  and  software  tools  for  evaluation  of  human-system  task  performance.  The 
HPBMS  facilitates  measurement  and  analysis  of  performance-related  variables  through  a 
distributed  network  of  collection  devices.  The  data  collection  network  operates  over 
standard  TCP/IP  protocols,  which  allows  for  easy  setup  of  both  local  and  remote 
collection  nodes  using  existing  local  area  networks,  wide  area  networks,  and  the  Internet. 

Performance  related  data  may  be  measured/collected  from  the  following  devices: 

•  Cybernet’ s  physiological  measurement  systems  (wearable/portable  data 
acquisition  systems) 

.  Eye  Tracking  Systems  with  Cybemet’s  Image  processing  software 

•  Cybernet’ s  Motion  Capture  (Body  Tracking)  System 

.  VCRs  with  RS232  interface  (for  video  and  audio  capture) 

•  Any  device  with  serial  data  output 

•  Any  collection  system  with  data  output  through  TCP/IP  network  socket 

•  Any  computer  application  with  data  output  through  TCP/IP  network  socket 
(provides  means  for  recording  performance  variables  during  HCI  task) 

Basic  operations  of  the  HPBMS  include: 

•  Interconnection  of  distributed  collection  nodes  over  network 
.  Synchronized  collection  of  data  over  all  nodes 

.  Real-time  viewing  of  any  data  from  any  node 

•  Synchronized  playback  of  post-collection  data  from  all  nodes 
.  Analysis  of  collected  data  using  built  in  functions 

.  Use  ofplug-in  functions  for  customized  on-line  analysis 

.  Event  marking  (tagging)  with  range-based  analysis  operations 
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Figure  37:  Block  Diagram  of  Human  Performance  Measurement  System  illustrating  the 
multiple  data  measurement  systems  integrated  into  a  comprehensive,  distributed,  data 

collection  and  analysis  architecture 


6.2  HPBMS  Applications  and  Usage 

The  HPBMS  is  a  comprehensive  and  highly  flexible  tool  intended  to  be  used  in  a  wide 
range  of  performance  assessment  and  human  factors  related  research.  A  researcher  may 
easily  configure  an  implementation  of  the  HPBMS  that  uses  any  number  of  collection 
nodes.  A  setup  may  use  one  or  more  of  the  Cybernet  developed  data  collection  systems, 
and/or  any  number  of  customized  data  collection  devices  that  have  serial  or  network 
socket  data  output.  Because  the  system  works  over  standard  TCP/IP  network  protocols 
the  data  collection  network  is  quickly  and  easily  established  over  existing  networks  and 
the  Internet,  allowing  remote  collection  anywhere. 


Example  applications  of  the  HPBMS  include: 

•  Study  of  Human  Computer  Interface  (HCI)  tasks 

•  Evaluation  and  design  of  improved  computer  interfaces 

•  Psycho-physiological  research 

•  Cognitive  research  and  assessment 

•  Posture,  gait  and  mobility  analysis 

•  Productivity  analysis  and  modeling 

.  Study  of  human  interaction  in  virtual  environments 
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Figure  38,  below,  illustrates  an  example  scenario  for  study  of  a  human  computer  interface 
(HCI)  task.  This  HPBMS  setup  may  be  grouped  into  two  functional  components:  1)  the 
User  Interface  Station,  and  2)  the  Investigator’s  Station.  The  User  Interface  Station  is 
where  the  subject  performs  the  HCI  task  and  performance-related  data  is  measured  using 
various  data  collection  devices.  The  data  collection  devices  are:  8  channel  and  16  channel 
physiological  monitors,  eye  tracker,  body  tracker,  and  video  camera. 


User  Interface  Station  Investigator  s  Station 


Wired  network  link 


VCR 

TV  Monitor 

VCR 


TV  Monitor 


Distributed  data  collection 
managed  by  the  Data  Collection 
Broker  (DCB)  software  application 
which  runs  on  each  collection  node 


Figure  38:  Example  Scenario  for  Study  of  Performance  in  HCI  Task 

The  Investigator’s  station  is  where  the  researcher  controls  and  monitors  the  data 
collection  and  experiment  process.  The  central  data  collection,  display,  and  analysis  PC 
provides  the  interface  for  initiating  data  collection,  as  well  as  viewing  and  analysis  of  the 
data.  The  eye  tracking  server  PC  performs  tiie  image  processing  algorithms  for 
computing  eye  position;  the  results  are  then  recorded  on  VCR  (gaze  location  indicated  on 
video  of  user’s  view  taken  from  eye  tracker)  and/or  recorded  through  the  DCB  software. 
A  second  VCR  and  TV  monitor  is  also  used  to  record  video  and  audio  from  an  optional 
video  camera  capturing  a  view  of  the  user  interface  station. 
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User  Interface  Station: 

•  User  Interface  PC 

•  Task  interfaces 

•  Keyboard,  mouse,  and  joystick  inputs 

•  Network  communications  to  a  DCB  node  for  recording  interface  events 
.  Eye  tracker  client  application  for  calibration  grid  display 

•  Physiological  measurement  systems 

•  Physiological  sensor  input  and  signal  conditioning 

.  DCB  software  for  networked  data  collection  and  communication 

•  Local  data  storage  medium 

•  Wired  or  wireless  networking 

•  User  Interface  Station  (continued): 

•  Eye  tracker 

•  Video  output  of  eye  to  Eye  Tracking  Server  PC 

•  Video  output  of  users  view  to  Eye  Tracking  Server  PC 
.  Video  camera 

•  Video  and  audio  output  of  user  interface  station  to  V CR 

•  Body  tracker 

.  Serial  communication  link  to  central  collection  node 

Investigator’s  Station; 

.  Central  data  collection,  display,  &  analysis  PC 

•  DCB  for  networked  data  collection  and  communication 

•  Data  collection  drivers  (serial  or  socket  communications)  to  Eye  Tracker  Server, 
Body  Tracker,  and  PC  controlled  VCRs 

•  Wireless  LAN  link 

.  Eye  tracker  client  application  for  control  operations 

•  Analysis  software  for  real  time  viewing  and  synchronized  playback 

•  Investigator’s  Station  (continued): 

•  Eye  Tracking  Server  PC 

•  Eye  tracker  server  application  (socket  comm,  link  to  the  other  PCs) 

.  2  channel  video  input  from  head-worn  eye  tracker  device 

.  Eye  tracking  display  and  video  output  to  V CR 
.  VCRs 

.  Serial  link  to  central  data  collection  PC  for  control 

•  Video  and  audio  input  from  camera  (V CR  1) 

•  Video  output  from  Eye  Tracking  Server  display  (VCR  2) 

•  TV  monitor  display 


6.3  HPBMS  Equipment  List 

Depending  on  the  specific  application,  the  HPBMS  will  be  configured  and  utilized 
differently.  The  following  provides  a  list  of  the  basic  components  of  the  HPBMS,  that 
may  be  included,  and  built  upon,  for  a  particular  application: 
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16  Channel  PPMS 
Typical  configuration: 

Network  card  (PCMCIA) 

Hard  drive  (PCMCIA) 

Electrode/sensor  cable 
Auxiliary  cable 
Power  converter 
Power  cable 
AC  power  supply 

8  Channel  PPMS 
Typical  configuration: 

Wireless  network  card  (PCMCIA) 

Flash  data  storage  (PCMCIA) 

Electrode/sensor  cable 
Auxiliary  cable 
Rechargeable  battery  pack(s) 

Battery  cable 

PPMS  Battery  Charger 

Eye  Tracking  System,  consisting  of: 

Head-worn  eye  tracking  device  (manufactured  by  ISCAN,  hic.  with  custom 
modifications) 

Eye  tracker  server  PC  with: 

Matrox  Meteor  frame  grabbers  (2) 

Video  scan  converter  (VGA  to  NTSC) 

Eye  tracking  software 

Video  camera  with  microphone  (optional) 

VCR  and  TV  monitor 

Central  Collection  and  Display/Analysis  PC,  with: 

Wireless  networking  card  (ISA)  and  antenna  (for  use  with  wireless  PPMS) 

Physiological  sensors,  which  may  include: 

EEG  cap  and  disposable  electrodes 
Electrodes  for  ECG,  EMG,  EOG,  etc. 

Respiration  band  (piezo-crystal  or  strain  guage) 

GSR  sensor  (optional) 

Temperature  sensor  (optional) 

Other  purchased  or  custom  sensors 

Electrode  lead  wires  and  splitters,  as  needed  for  sensor  inputs 


77 


CYBERNET  SYSTEMS  CORPORATION 


Event  marking  switch  (5  volt) 

DCAE  software  (HPMS  Installation  CD) 

Body  Tracker  System  (optional) 

Other  items  that  may  be  necessary  include  rubbing  alcohol  for  application  of  electrodes,  a 
static  mat  (to  be  placed  under  the  subject’s  seat)  to  minimize  noise  induced  in  lead  wires, 
and  a  surge  protector  for  all  AC  powered  equipment. 


6.4  List  of  HPBMS-Related  Documentation 

•  HPBMS  System  Overview 

This  document  describes  the  high  level  functionality  of  the  Human  Performance-Based 
Measurement  System  and  general  usage.  The  HPBMS  subsystems  are  introduced  in  this 
tnannal,  however,  operational  details  of  the  individual  subsystems  are  addressed  in 
separate  documents,  as  described  next. 

•  Data  Collection  and  Analysis  Environment  (DCAE)  Software  Documentation 

The  DCAE  software  is  instrumental  to  the  operation  of  the  Human  Performance  Based 
Measurement  system.  It  coordinates  all  collection,  storage,  transmission,  display,  and 
analysis  of  measured  data.  Documentation  for  the  DCAE  software  consists  of  3  separate 
documents: 

DCAE  System  Overview 

This  document  contains  a  general  overview  of  the  entire  DCAE  system.  It 
describes  the  architecture  and  capabilities  of  the  two  primary  components  of  the 
system  -  the  Data  Collection  Broker  (DCB)  and  the  Analysis  and  Playback 
System  (APS).  The  DCB  controls  the  acquisition,  communication,  and 
management  of  data  over  the  distributed  collection  network,  while  the  APS 
provides  tools  for  viewing  and  analysis  of  the  data. 

DCB  User’s  Guide 

This  document  describes  the  user  interface  for  the  Data  Collection  Broker 
(DCB)  portion  of  the  DCAE  software.  It  details  both  the  command  line  interface 
and  the  Graphical  User  Interface  (GUI),  either  of  which  may  be  used  to  control 
networked  data  collection  operations.  Topics  covered  include  system 
configuration,  user-level  commands,  scripting,  and  how  to  run  a  simple  data 
collection. 

APS  User’s  Guide 

This  document  describes  the  user  interface  to  the  Analysis  and  Playback 
System  (APS).  It  guides  the  user  through  the  menu  operations  and  describes  each 
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of  the  data  viewing  and  analysis  functions.  It  also  illustrates  the  real-time  viewing 
and  synchronized  playback  capabilities. 

•  Physiological  Measurement  Systems  User’s  Guide 

This  document  describes  configuration  and  use  of  the  physiological  measurement 
systems.  It  provides  technical  specifications  for  both  the  8  channel  (wearable)  system  and 
the  16  channel  (portable)  system.  These  devices  may  be  used  independently  or  in 
conjimction  with  other  HPBMS  data  collection  devices. 

•  Eye  Tracking  System  User’s  Guide 

This  document  describes  use  of  the  eye  tracking  system.  It  addresses  both  independent 
operation  and  operation  within  the  HPBMS  data  collection  environment.  The  eye  tracking 
system  is  based  on  a  video  processing  technique  that  produces  gaze  location  within  the 
user’s  field  of  view  (captured  with  a  head  mounted  camera)  or  within  a  viewed  computer 
display. 

•  Firefly  User’s  Guide  (Body  Tracker  -  Optional  Subsystem) 

This  is  the  user’s  guide  for  the  Firefly  Body  Tracking  system.  It  addresses  both 
independent  operation  and  operation  within  the  HPBMS  data  collection  environment. 

The  Firefly  uses  three  cameras  to  track  the  location  of  infra-red  tags  (LEDs)  in  three- 
dimensional  space.  These  tags  can  be  placed  on  a  person’s  body  to  track  limb  positions, 
on  objects  to  track  their  motion,  or  anything  which  you  want  to  collect  exact  position  data 
about. 

•  Tips  &  Troubleshooting  Guides 

This  document  pair  covers  all  portions  of  the  Human  Performance  Measurement  System. 
They  contain  useful  shortcuts,  important  reminders,  and  tips  and  tricks  to  achieve  better 
performance.  The  Troubleshooting  Guide  contains  a  list  of  possible  errors  and  problems 
which  might  occur  while  using  the  system,  and  how  to  correct  them. 


6.5  The  DCAE  Software 
6.5.1  Introduction 

The  distributed  Data  Collection  and  Analysis  Environment  (DCAE)  is  composed  of  four 
parts  that  work  together  to  make  up  a  comprehensive  collection,  test,  analysis,  and 
validation  suite.  TTiese  four  parts  are: 

DCB  The  Data  Collection  Broker  is  a  multi-platform  console  and  Java  GUI 

based  application  which  is  the  backbone  of  all  data  collection  operations. 
This  system  is  capable  of  controlling  and  collecting  data  from  a  variety  of 
low-level  hardware  devices  over  a  heterogeneous  distributed  network. 
This  is  the  starting  point  for  all  data  collections. 
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DCB  DCB  Drivers  are  loadable  modules  which  control  the  collection  of  data 

Drivers  from  hardware  and  virtual  devices.  These  drivers  are  responsible  for 

parsing  and  decommutating  the  data  received  by  the  device  into  an  object 
oriented  data  framework. 

APS  The  Analysis  and  Playback  System  is  the  primary  method  for  monitoring, 
replaying,  analyzing,  manipulating,  and  exporting  data  collected  using  the 
DCAE. 

Analysis  A  toolbox  extends  the  standard  functions  and  visualization  tools  to 
Toolboxes  support  advanced  data  analysis.  These  toolboxes  can  either  be  pre¬ 
packaged  or  custom-programmed  based  on  customer  needs. 

The  system  block  diagram  below  shows  the  integration  of  the  parts  mentioned  above  into 
the  DCAE  System. 


Toolboxes  are  plug-ins  which 
provide  custom  functions  and 
viewers  to  extend  the  APS . 

The  Data  Interface  Library 
provides  a  standard,  open  API  to 
handle  streams  of  fixed-  and 
variable-size  data.  It  is 
implemented  using  shared 
memory  to  give  many 
applications  seamless  and  timely 
access  to  its  data. 

Real-Time  Modules 
process  data  in  the  DIL  to  create 
new  data.  These  modules  can 
run  on  any  node  which  is 
running  the  DCB,  allowing  for 
distributed  computing. 

Custom  Data  Types  support 
specific  drivers  (i.e.  an  MPEG 
data  type  for  full-  motion  video). 

Drivers  plug  in  to  the  DCB  to 
provide  support  for  hardware 
devices.  Examples  include 
VCR,  Eye  Tracker  and  serial 
port  drivers. 
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All  of  the  plug-in  modules  (drawn  slightly  displaced  from  the  container  module)  are 
implemented  as  Dynamic  Link  Libraries  (in  Windows)  or  shared  libraries  (in  UNIX). 
This  allows  the  modules  to  be  loaded  at  run-time  for  seamless  integration  with  host 
applications. 

All  of  the  applications  interact  through  the  Data  Interface  Library.  Any  combination  of 
applications  can  be  used.  The  applications  can  start  up  and  shut  down  independently  of 
one  another. 


6.5.2  List  of  DCAE  Software  Documentation 

DCAE  System  Overview 

This  document  contains  a  general  overview  of  the  entire  Data  Collection  and  Analysis 
Environment  (DCAE)  system.  It  describes  the  architecture  and  capabilities  of  the  two 
primary  components  of  the  system  -  the  Data  Collection  Broker  (DCB)  and  the  Analysis 
and  Playback  System  (APS). 

DCB  User ’s  Guide 

This  document  describes  the  user  interface  for  the  Data  Collection  Broker  (DCB).  It 
covers  system  configuration,  system  inputs,  user-level  commands  and  scripting,  as  well  as 
the  graphical  user  interface  (GUI).  A  “getting  started”  section  is  included  which  describes 
the  necessary  steps  for  a  simple  data  collection  using  the  GUI.  It  also  details  a  DCB 
session  which  guides  the  user  through  a  sample  data  collection  using  the  command  line 
interface. 

APS  User’s  Guide 

This  document  describes  the  user  interface  to  the  Analysis  and  Playback  System  (APS). 

It  guides  the  user  through  menus  and  describes  each  available  data  viewer  and  analysis 
function. 

Tips  &  Troubleshooting  Guides 

While  these  documents  cover  all  portions  of  the  Human  Performance  Measurement 
System,  they  also  include  extensive  sections  on  the  DCB  GUI,  DCB  Console,  and  APS. 
Refer  to  them  for  help  with  any  problems  that  may  occur  while  using  the  DCAE,  and  for 
helpful  tips  and  suggestions. 


6.5.3  Description  of  the  Data  Collection  Broker  (DCB) 

The  DCB  is  the  primary  software  component  and  interface  that  coordinates  all  data 
collection  operations  within  the  DCAE  architecture.  It  captures  synchronized  data  from 
networked  collection  devices  that  run  generic  or  custom  acquisition  drivers  that  operate 
within  the  DCB.  It  has  the  following  characteristics; 
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•  Works  on  a  variety  of  operating  systems  including  DOS,  UNDC  (FreeBSD),  and 
Windows  NT. 

.  Allows  simultaneous  data  collection  from  multiple  devices.  Each  device 
represents  a  single  stream  of  data  (from  a  physiological  monitor,  eye  tracking 
system,  digital  VCR,  etc.). 

.  Can  be  used  to  communicate  with  other  remote  nodes  to  control,  collect,  and/or 
stimulate  devices  connected  to  them.  Data  collected  from  the  remote  nodes  can 
be  monitored  in  real  time  (actual  rate  depends  on  connection  bandwidth). 

.  Supports  multiple  data  types  including  float,  double,  long  integer,  integer, 
boolean,  graphic,  and  string  types. 

.  Can  archive  collected  data  to  the  local  computer.  The  Analysis  and  Playback 
System  can  then  be  used  to  replay  the  collected  data. 

.  Can  use  scripts  to  configure,  control,  and  execute  pre-arranged  data  collections. 

.  Can  incorporate  real-time  modules  which  support  concunent  real-time  processing 
during  data  collection.  Some  examples  of  real-time  modules  are;  Expert  Systems, 
Neural  Networks,  Limit  Checking  Alerts,  and  other  custom  functions. 


Figure  39:  Example  of  a  DCB  Network  Configuration 

The  DCB  is  operable  in  two  different  forms:  a  DOS-like  console  application  (the  DCB 
Console),  or  a  Graphical  User  Interface  application  (the  DCB  GUI).  The  figures  below 
illustrate  these  applications.  For  more  detail  on  these  software  applications,  refer  to  the 
DCB  User’s  Guide. 

List  of  Developed  DCB  Drivers 

Currently  the  DCB  supports  the  following  collection  devices: 
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Serial  Driver  .  i  •  *  ^ru 

This  is  a  generic  interface  for  collecting  any  data  off  of  a  serial  interlace.  Ine 

packet  format  is  defined  in  the  configuration  and  then  the  driver  can  collect  that 
data. 

mis  f generic  interface  for  collecting  any  data  through  ne^ork  socket  link.  The 
packet  format  is  defined  in  the  configuration  and  Aen  the  driver  can  collect  that 

data. 

PPMS  Driver 

This  is  the  driver  that  performs  data  collection  operations  on  the  portable 
physiological  measurement  system  (PPMS).  The  PPMS  unit  and  its  driver  extend 
the  Data  Collection  Broker  (DCB)  to  perform  multi-channel  acquisition  and 
analysis  of  physiological  signals.  The  PPMS  unit  is  a  portable  DCB  node 
specialized  for  use  as  a  data-acquisition  engine.  The  PPMS  Dnver  interfaces  the 
PPMS  hardware  to  the  DCB  system,  which  provides  data-management 
functionality,  network  connectivity,  and  time  synchronization  of  multiple  DCB 
nodes.  Together,  the  hardware  and  software  combine  to  form  a  complete  solution 
for  the  collection  of  physiological  data  under  local  or  remote  control. 

Pulse  Oximetry  Driver 

The  Pulse  Oximetry  Driver  interfaces  the  DCB  to  a  pulse  oximetry  module 
manufactured  by  the  Nonin  Corporation,  providing  plethysmograph,  blood 
oxygenation,  and  pulse  rate  data.  Typically  this  module  is  operated  through  the 
serial  port  interface  of  the  PPMS  device,  but  may  also  operate  independently. 

Blood  Pressure  Cuff  Driver  ,  ui  .1 

The  Blood  Pressure  Cuff  Driver  provides  an  interface  to  an  automated  blood 
pressure  cuff  via  either  RS-232  or  IRDA  communication  modes.  Typically  this 
module  is  operated  through  the  PPMS  device,  but  may  also  operate  independently. 

This  driver  provides  an  interface  for  the  control  of  a  digital  VCR  with  serial  port 
interface,  to  provide  time-synchronized  collection  and  playback  video  data. 

EyeTracker  Driver  . 

This  driver  receives  data  from  Cybemet’s  eye  tracking  system,  using  network 

socket  communications. 

BodyTracker  Driver  . 

This  driver  receives  data  from  Cybemet’s  body  tracking  (motion  capture)  system, 

using  serial  port  communications. 


TestWave  Driver 
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This  driver  generates  pairs  of  sine  and  cosine  waves  for  testing  and  validation 
piuposes.  Amplitude,  frequency,  and  number  of  pairs  generated  are  among  the 
parameters  which  can  be  manipulated. 

6.5.4  Description  of  the  Analysis  and  Playback  System 
The  APS  is  the  primary  software  component  and  interface  that  coordinates  all  data 
analysis,  manipulation,  and  viewing  within  the  DCAE  architecture.  It  is  a  Windows  based 
interface  that  has  the  following  features: 

•  Can  create  views  of  sets  of  data  fields.  Strip  charts,  bar  charts,  text  views,  and 
other  standard  views  are  supported.  Toolboxes  can  be  used  to  provide  additional 
or  customized  views. 

•  Can  coordinate  and  synchronize  playback  of  data  displayed  in  the  opened  views. 

.  Can  apply  functions  to  the  data  fields  to  create  new  data  fields.  Some  supported 

functions  are  filters,  FFT’s,  and  smoothing  and  resampling  operations.  Toolboxes 
can  be  used  to  provide  additional  or  customized  functions. 

•  Allows  you  to  manipulate  the  data  fields.  You  can  cut,  copy,  and  paste  fields  or 
sections  of  fields  to  move  them  from  one  collection  set  to  another. 

•  Allows  you  to  annotate  and  select  data  of  interest  for  further  analysis.  This  is 
done  by  creating  tag  fields.  The  tag  fields  are  used  to  mark  and  record  user 
observations  made  while  analyzing  the  data,  and  can  be  used  to  define  ranges  for 
playback  and  function  execution. 

•  Allows  workspaces  to  be  created,  saved,  and  restored.  This  allows  for  the 
automatic  restoration  of  a  set  of  views  and  their  attributes  (position,  size,  scaling 
properties)  when  a  collection  set  is  loaded. 

.  Can  monitor  data  collected  from  the  DCB  in  real  time,  automatically  adapting  to 
network  bandwidth  constraints. 

.  Can  export  data  for  further  analysis  and/or  display  using  third  party  software. 

Figure  40  shows  a  sample  Analysis  workspace  configuration,  demonstrating  EEC  signal 
and  topographic  display;  numeric  and  text  viewers  for  ECG,  respiration,  and  EMG 
signals;  a  VCR  control  viewer  window;  and  synchronized  playback  controls.  For  more 
details  on  operation  of  the  Analysis  and  Playback  System  software,  refer  to  the  APS 
User’s  Guide. 
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Figure  40:  Analysis  and  Playback  System  Sample  Workspace  Screen  Shot 


Available  Analysis  Viewers 

Following  is  a  list  of  currently  available  viewers  within  the  APS  interface: 

•  VCR  Viewer 

•  Text  Viewer 

•  Tag  EditorA^iewer 

•  Spectrum  Viewer 

•  Numeric  Viewer 

.  Multi-channel  Stripchart  Viewer 

•  Evoked  Potential  Viewer 
.  EEG  Brain  Map  Viewer 

.  Motion  Capture  (Body  Tracker)  Viewer 

•  Alarm  Viewer 

Analysis  Functions  .  ^  •  n  *  j 

Following  is  a  list  of  the  currently  available  analysis  functions,  for  processing  collected 

data,  within  the  APS  interface. 

.  Scientific  Calculator  Operations 

•  FIR  (Finite  Impulse  Response)  Filter 

•  Moving  Window  Average 

•  Median  Filter 
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•  Noise  Generation 

•  Rate  Detection 

•  RMS  Power 

.  State  Classification  (threshold  detection) 

•  Artifact  detection 

•  Statistical  Operations  (via  Tag  Viewer) 

Data  Export 

The  APS  interface  also  allows  the  user  to  export  data  to  a  text  file  (tab,  comm^  or  space 
delimited)  or  to  a  Matlab  format  file  for  further  data  analysis  and/or  manipulation. 


6.6  Physiological  Measurement  Systems 
6.6.1  Overview 

The  Portable  Physiological  Measurement  System  (PPMS)  is  a  miniature  computing 
platform  that  performs  multi-channel  data  collection  and  provides  ne^ork 
interoperability.  The  system  operates  within  Cybemet’s  Data  Collection  and  Analysis 
Environment  (DCAE)  architecture  for  distributed  data  collection  over  networks  (see 
separate  DCAE  manuals). 

Programmable  signal  conditioning  features  of  the  PPMS  allow  for  measurement  of  any 
signal  ranging  from  1  uV  to  +5  V,  and  provides  selectable  filtering  and  other  conditioning 
operations.  The  system  has  a  PCMCIA  interface  providing  convenient  hardware 
configuration  using  PC-cards  for  data  storage  (hard  drives,  FLASH  cards),  networking 
(Ethernet,  wireless  LAN),  and  other  functions. 


Currently  two  versions  of  the  physiological  measurement  system  are  available;  a  16- 
channel  device,  and  an  8-channel  device. 


AC  POWER  SUPPLY 


Power  Battery 

Converter  Sub-Packs 

Module 
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Software  for  the  physiological  measurement  system  is  based  on  the  DCAE  system.  The 
physiological  measurement  system  runs  a  DOS  version  of  the  Data  Collection  Broker. 

All  remote  operability  and  interaction  is  achieved  through  the  DCB  interface,  and  remote 
viewing  and  analysis  of  data  can  be  performed  through  the  Analysis  and  Playback  (APS) 
software. 


The  system  may  be  operated  through  three  different  methods:  1)  the  front  panel  interface 
(16  channel  system  only),  2)  the  DCB  console  running  on  a  networked  machine,  or  3)  the 
DCB  graphical  user  interface  running  on  a  networked  machine.  The  following  sections 
provide  a  description  of  the  PPMS  devices  and  some  examples  of  operational  use.  For 
more  details  on  configuration  and  operation  of  these  devices,  refer  to  the  PPMS  Users 
Guide  document. 


6.6.2  Hardware  Descriptions 

6.6.2.1  The  16-Channel  Device  Specification 


Cybemet’s  network-operable  physiological  measurement  system  is  a  portable  data 
collection  device  that  supports  a  wide  range  of  physiological  and  other  signal 
measurements.  It  is  a  miniature  PC-based  system  that  offers  extensive  programmability 
and  flexibility.  Uniquely  designed  signal  conditioning  circuitry  allows  for  connection  of 
any  channel  to  almost  any  sensor  type,  including  biopotential  electrodes  for  ECG,  EMG, 
EEG,  etc.  The  system  has  2  serial  communications  interfaces  as  well  as  4  PC-card  slots 
for  data  storage  (hard  drive  or  flash  memory),  networking  (wired  or  wireless),  and  other 
functions  (such  as  global  positioning).  Integrated  blood  pressure  (NIBP)  and  pulse 
oximetry  modules  are  also  available.  For  applications  requiring  more  than  1 6  channels, 
multiple  devices  may  be  used  simultaneously. 
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The  16-Chaimel  PPMS  is  designed  to  be  as  light  as  possible  while  still  allowing  for  16 
channels  on  which  to  collect  data.  It  is  ideally  suited  for  use  as  a  portable  table-top 
machine,  although  it  is  small  and  light  enough  to  be  worn  on  a  belt  or  carried  in  a 

backpack. 

J6  Channel  PPMS  Technical  Specifications: 


Physical  Dimensions  (approx.): 

4”  X  5.5”  X  8”,  contoured 

Weight  (approx.): 

3.5  lbs.  (approx.) 

Network  Protocol: 

TCP/IP  (Ethernet  or  wireless) 

Signal  Conditioning 

Amplification: 

Differential' 

Isolation  (channel  to  channel): 

500  volts 

Isolation  (channel  to  system): 

1,500  volts 

Input  noise  (gain  >  10,000): 

<  1  pV  RMS 

Input  voltage  range: 

■t"/-5  V 

CMRR: 

>107  dB  @  60Hz 

Gain: 

1  to  300,000 

Coupling: 

AC  or  DC 

High  pass  Filter: 

4-pole,  4  frequency  options 

Baseline  restoration: 

Programmable  on/off 

60  Hz  notch  filter: 

Programmable  in/out 

Low  pass  filter: 

4-pole,  8  frequency  options 

DC  offset: 

-5  volts  to  +5  volts 

A/D  Conversion: 

12  bit  resolution,  10  volt  range 

Power  Supply: 

AC,  or  Battery  Pack  (6-15  volts) 

PC  Card: 

3  available  slots 

Data  Storage: 

Up  to  2.08  Gbytes 

LCD  Display: 

128  X  64  graphical,  w/  backlight 

User  Input: 

8  button 

Auxiliary  Interfaces: 

Two  RS-232  serial  ports 

16  Channel  PPMS  Hardware  Confieuration: 

The  16  channel  PPMS  device  is  pre-configured  for  the  typical  operational  scenario, 
however  the  flexible  platform  allows  for  configuration  of  the  hardware  to  meet  specific 
application  requirements. 

Please  note:  if  you  wish  to  reconfigure  the  PPMS  hardware  it  is  strongly  recommended 
that  you  contact  Cybernet  technical  staff  and  arrange  for  proper  handling.  Improper 
handling  of  the  internal  hardware  components  can  damage  the  device. 

The  hardware  is  already  configured  to  meet  the  needs  of  most  applications,  however  you 
may  wish  to  configure  it  with  different  PC  Cards  to  meet  the  particular  needs  of  your 
application.  For  this  purpose,  the  back  end  of  the  PPMS  is  removable.  First,  disconnect 
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any  ethemet  attachments  that  exit  the  PPMS  through  the  rectangular  slot  on  the  back,  and 
all  other  cables  (battery  cables,  patient  cables,  etc. . To  remove  the  back  cap,  press  in 
on  the  short  sides,  then  slowly  pull  the  cap  away  from  the  main  body  of  the  unit  while 
continuing  to  keep  pressure  on  the  sides  of  the  cap.  There  are  2  clips  on  each  side  of  the 
cap  which  must  be  pressed  on  in  order  to  remove  the  cap. 


Looking  directly  at  the  back  of  the  open  16-Channel  PPMS,  the  interior  of  the  unit  is 
arranged  as  follows  (from  left  to  right): 


R  PrApratnmabIft  Amplifier  Boards  -  each  board  supports  2  channels  of  data 
independently.  The  boards  are  numbered  1-8,  starting  with  8  at  the  left.  Board  8  supports 
channels  15  and  16,  Board  7  supports  channels  13  and  14,  etc. 


4  PCMCIA  Slots  (2  carrier  boards  with  dual  slots’)  -  In  the  default  configuration  a  hard 
drive  (520  MByte)  and  Ethemet  networking  card  are  currently  located  in  the  PCMCIA 
(PC  Card)  slots.  Two  slots  are  potentially  available  for  additional  PC  Cards  if  desired  for 
your  application  -  these  may  be  used  for  additional  data  storage  or  other  functions.  Some 
PC  Cards  (such  as  the  hard  drive)  require  the  space  of  two  slots  and  may  limit  the  number 
of  Cards  that  may  be  used.  Additional  space  is  available  between  the  rightmost  slot  and 
the  CPU  carrier  board  -  double  width  Cards  should  therefore  be  placed  in  the  rightmost 
slot  when  trying  to  maximize  the  number  of  PC  Cards  that  may  be  used. 


CPU  Carrier  Board  -  This  board  contains  the  CPU  and  associated  hardware.  The  system 
may  be  configured  with  a  386  or  486  based  processor  unit  depending  on  the  desired 
tradeoff  between  power  consumption  and  performance.  The  default  configuration 
contains  the  486  processor.  If  it  is  deemed  necessary  to  change  the  CPU  configuration 
please  contact  Cybernet  technical  staff  for  more  information. 
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6.6.2.2  8"Channel  Device  Specification 


Figure  43:  The  8  Channel  Portable  Physiological  Measurement  System 


Cybemet’s  wearable  physiological  measurement  system  is  a  smaller,  lighter,  lower-power 
version  of  the  16  channel  physiological  monitor  described  above.  It  has  the  same 
programmable  functionality  of  the  16  channel  system,  except  for  fewer  channels,  fewer 
PC-card  slots,  and  a  less  extensive  user  interface. 


R  Channel  PPMS  Technical  Specifications: 

Physical  Dimensions  (approx.): 
Weight  (approx.): 

Network  Protocol: 

Signal  Conditioning 
Amplification: 

Isolation  (channel  to  channel): 
Isolation  (channel  to  system): 
Input  noise  (gain  >  10,000): 
Input  voltage  range: 

CMRR: 

Gain: 

Coupling: 

High  pass  Filter: 

Baseline  restoration: 

60  Hz  notch  filter: 

Low  pass  filter: 

DC  offset: 

A/D  Conversion: 

Power  Supply: 


2.6”  X  5.7”  X  6.5”,  contoured 
1.9  lbs.  (without  battery) 
TCP/IP  (Ethernet  or  wireless) 

Differential 
500  volts 
1,500  volts 
<  1  pV  RMS 
+/.5V 

>107  dB  @  60Hz 
1  to  300,000 
AC  or  DC 

4-pole,  four  frequency  options 
Programmable  on/off 
Programmable  in/out 
4-pole,  eight  frequency  options 
-5  volts  to  +5  volts 
16  bit  resolution,  20  volt  range 
AC,  or  Battery  Pack  (6- 15V) 
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PC  Card:  2  available  slots 

Data  Storage:  Up  to  2.08  Gbytes 

User  Control:  Power-up  and  shutdown 

8  Channel  PPMS  Hardware  Confieuration: 

The  8  channel  PPMS  device  is  pre-configured  for  the  typical  operational  scenario, 
however  the  flexible  platform  allows  for  configuration  of  tiie  hardware  to  meet,  specific 
application  requirements. 

Please  note:  if  you  wish  to  reconfigure  the  PPMS  hardware  it  is  strongly  recommended 
that  you  contact  Cybernet  technical  staff  and  arrange  for  proper  handling.  Improper 
handling  of  the  internal  hardware  components  can  damage  the  device. 

The  hardware  is  already  configured  to  meet  the  needs  of  most  applications,  however  you 
may  wish  to  configure  it  with  different  PC  Cards  to  meet  the  particular  needs  of  your 
application.  For  this  purpose,  the  back  end  of  the  PPMS  is  removable.  First,  disconnect 
any  ethemet  attachments  that  exit  the  PPMS  through  the  rectangular  slot  on  the  back,  ^d 
all  other  cables  (battery  cables,  patient  cables,  etc...).  To  remove  the  back  cap,  you  will 
need  a  hex  wrench  to  unscrew  the  three  silver  screws  holding  the  back  cap  on.  It  is 
suggested  that  when  removing  the  back,  you  tilt  the  PPMS  up  so  that  it  is  resting  with  the 
fi-ont  face  down. 


Figure  44:  View  of  the  8  Channel  PPMS  Hardware  Components  Inside  Back  Panel 

Looking  directly  at  the  back  of  the  open  8-channel  unit,  the  boards  are  arranged  as 
follows: 

From  top  left  to  bottom  left: 
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CPU  Carrier  Board  -  the  default  configuration  uses  a  486  based  processor 

2  PCMCIA  slots  ri  board  with  dual  slot)  -  in  the  default  configuration  the  two 
slots  are  occupied  by  an  85MByte  Flash  (data  storage)  card  and  a  Proxim 
RangeLAN2  wireless  networking  card 

From  top  right  to  bottom  right: 

d  Prnprflmmable  Amplifier  Boards  -  each  board  supports  2  channels  of  data 
independently. 

Do  not  remove  any  of  the  boards  without  first  consulting  Cybernet  technical  staff.  The 
PC  Cards  may  be  removed  and  replaced,  however  power  should  first  be  turned  off  and 
all  cables  disconnected. 

6.6.2.3  The  Patient  Cable 

The  Patient  Cable  is  the  means  by  which  physiological  signals  are  captured  and  recorded 
by  the  PPMS.  There  are  several  different  versions  of  the  cable,  but  they  all  follow  the 
same  basic  design.  At  one  end  is  a  plug  for  connecting  the  cable  to  the  PPMS.  The  other 
end  contains  a  number  of  electrode  pairs  (8  or  16  pairs,  depending  on  which  PPMS  you 
are  using)  and  connections  for  electrical  grounding.  Each  of  the  pairs  is  numbered  -  this 
number  corresponds  to  the  PPMS  channel  to  which  it  is  linked.  Each  pair  consists  of  a 
black  and  a  red  connector  pin:  red  is  the  positive  input,  black  is  the  negative  input. 


Figure  45:  View  of  the  Standard  Patient  Cables  for  (a)  the  16  Channel  Device,  and  (b) 

the  8  Channel  Device 


The  16-Channel  patient  cable  has  a  rectangular  clip  at  the  PPMS  end.  The  electrode  leads 
are  paired  off  as  normal.  There  is  one  shared  ground  for  every  4  channels  and  which  is 
bundled  with  its  respective  group:  channels  1-4, 5-8, 9-12, 13-16. 
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The  S-Channel  patient  cable  has  a  circular  L-coimector  at  the  PPMS  end.  The  electrode 
leads  are  then  paired  off  as  normal.  There  is  one  common  ground  (green  vore)  for 
channels  1  -  6.  Channels  7  and  8  each  have  independent  grounds  that  are  bundled  with 

the  respective  inputs. 


The  connectors  at  the  end  of  the  paUent  cable  are  male,  and  will  accept  the  female 

connector  ends  of  Standard  medical  electrodes. 


6.6.2.4  Additional  Components 


6.6.3  The  PPMS  Front  Panel  Interface 

The  front  panel  local  menu  interface  is  available  only  for  the  16  ch^el  version  of  &e 
system.  TOs  interface  provides  complete  control  over  operation  of  the  system  (for  the  8 
channel  device,  control  must  be  performed  through  remote  operation  using  the 
command  console  or  the  DCB  graphical  user  interface). 


MENU/ 


The  menu  system  is  composed  of  three  main  elements.  The  user  can  select  the  v^ous 
options  using  the  five  buttons  above,  the  two  buttons  to  the  right,  and  the  single  button  to 
the  left  of  the  display.  The  five  buttons  above  the  display  provide  menu  selections  as  well 
as  text  and  numeric  manipulation  functions.  The  two  buttons  to  the  right  of  the  display 
provide  the  ability  to  scroll  up  or  down  through  a  list  of  choices.  The  single  button  to  the 
left  of  the  display  is  often  utilized  as  an  “OK”  button  to  commit  to  changes.  An  example 
menu  illustrating  the  various  button  functions  is  shown  below. 


The  example  menu  shown  above  allows  the  user  to  set  the  gain  of  one  or  all  of  Ae 
physiological  channels  in  the  PPMS.  The  row  of  icons  at  the  top  of  the  display  identifies 
the  function  of  each  of  the  five  buttons  located  above  the  display.  The  buttons  labeled  ^ 
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and  allow  the  user  to  move  the  cursor  left  and  right,  respectively,  for  example,  to 
select  a  digit  to  modify.  The  buttons  labeled  +  and  -  allow  the  user  to  increment  and 
decrement,  respectively,  either  numeric  or  alphabetic  fields.  The  EXIT  button  allows  the 
user  to  return  to  the  previous  menu  without  making  any  changes.  The  two  buttons  to  the 
right  of  the  display  labeled  and  nI'  allow  the  user  to  scroll  up  and  down,  respectively, 
through  the  displayed  list  of  options.  The  button  to  the  left  of  the  display  labeled  OK 
allows  the  user  to  commit  to  the  changes,  thus  altering  the  configuration,  and  return  to  the 
previous  menu.  The  numeric  entry  editor,  for  modifying  parameters  which  can.be  set  to  a 
range  of  arbitrary  values,  works  as  follows:  when  the  left  or  right  arrow  is  pressed,  it  will 
move  to  an  existing  digit  if  there  is  one,  otherwise  it  wraps  arotmd  to  the  other  end.  If  the 
decimal  point  is  selected,  the  +  (increment)  or  -  (decrement)  buttons  will  slide  the 
decimal  point  left  or  right.  If  the  leftmost  digit  is  highlighted  and  incremented  past  9,  the 
result  is  a  "carry"  (i.e.  a  new  leftmost  digit  appears  set  to  .l).  Decrement  of  most 
significant  digit  (MSD)  of  numeric  parameters  is  reduced  to  the  next  smaller  place  value 
when  the  MSD  is  1 . 

All  the  necessary  functionality  of  the  PPMS  can  be  accessed  through  this  menu  system. 
The  user  can  choose  to  configure  the  PPMS  driver,  start  and  stop  a  collection,  view 
single-channel  stripcharts  or  multi-channel  bar  charts,  set  various  system  parameters,  or 
shut  down  the  PPMS  unit.  The  figure  below  illustrates  ftie  top-level  view  of  the  PPMS 
menu  system. 


6.6.4  Quick  Start  Operation  Using  the  Front  Panel 

The  recommended  method  for  PPMS  operation  is  by  using  the  DCBGUI  application  (see 
Section  6.6.4),  however  the  front  panel  can  be  extremely  useful  for  some  situations. 
Following  is  an  excerpt  firom  the  PPMS  User’s  Guide  that  provides  a  sample  operation 
scenario  for  PPMS  data  collection  using  the  front  panel  interface  of  the  16  channel 
device. 
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(1)  Turn  on  the  PPMS  by  pressing  the  “ON/Backlight”  button. 

(2)  In  approximately  25  seconds  the  computer  will  completely  boot  up.  Once  the  boot  up 
is  complete  the  Cybernet  Systems  Lx)go  will  appear  followed  by  the  PPMS  Status  menu. 


MENU 


PPMS  Status 

Config:  UNTITLED.INI 
Stream:  TESTOOOO 
Status:  Ready 


(3)  Press  the  “MENU/EXIT”  button  above  the  display  labeled  MENU  to  get  into  the 


PPMS  Main  Menu. 


PPMS  Main  Menu 
Configuration 
Coliection 
'iew _ 


XH 


Vie 

IS! 

Shutdown 


4. 


(4)  Your  current  selection  will  be  Configuration.  Select  System  by  using  the  up/down 
buttons  and  press  the  red  OK  button. 

(5)  You  will  now  be  at  the  Set  System  Params  menu.  Highlight  the  selection  you  wish 
to  modify  and  press  OK.  You  will  probably  need  to  change  or  create  the  Data  Directory, 
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Time,  Date,  Node  Name,  and  IP  Address  if  this  is  the  first  time  you  are  using  the  PPMS. 
For  a  more  detailed  explanation  of  these  menu  options,  consult  the  Front  Panel 
Operations  Section  of  this  manual. 


XH 


Set  System  Params 

Time 

Dote 

Node  Nome 
IP  Address 
Console  Access 
Display 


When  you  have  finished  correctly  setting  the  system  parameters,  press  the  exit  button 
from  the  Set  System  Params  menu  to  return  to  the  PPMS  Main  Menu. 

(6)  The  PPMS  will  already  have  a  default  .ini  file  loaded  (as  indicated  on  the  PPMS 
Status  Screen).  For  more  information  on  changing  this  .ini  file,  selecting  a  different  .ini 
file,  or  creating  a  new  .ini  file,  refer  to  Section  7.5. 

(7)  From  the  PPMS  Main  Menu  highlight  Collection  from  the  list  and  press  OK. 


0 

K 


PPMS  Main  Menu 


Xehit 


Configuration 


Collection 


View 

System 

Shutdown 


t 

I 


(8)  You  will  now  be  at  the  Collect  Data  menu.  Enter  name  of  file  to  store  data.  Make 
sure  file  name  does  not  already  exist;  if  it  does  select  another  name. 


^  DIGIT  4  I  +  VALUE  “  |X  EKh 


Collect  Data 


Name:  TESTOOOO 


start 

Stop 


(9)  Then  highlight  Start  from  Collect  Data  menu  and  press  OK.  The  Start  Options 
menu  has  'Start  Now’ ,  'Start  in  10  seconds’  and  'Start  at  —  '  options.  The  Start  at  time 
should  default  to  the  time  previously  set.  To  start  immediately  without  a  delay,  highlight 
start  now  to  start  data  collection.  Press  OK. 


97 


CYBERNET  SYSTEMS  CORPORATION 


^  DIGIT  ^  1  +  ypLuE  -  lx  EMIT 


Start  Options 


start  Now 


Start  in  XX  seconds 


n  Start  at  HH:MM:SS 


(10)  If  collection  can  startup  you  will  be  returned  to  the  PPMS  Status  menu.  Status: 
Ready  will  change  to  Status:  Collecting  once  the  PPMS  has  started  collecting.  • 


MENU 


PPMS  Status 

Config:  UNTITLED.INI 
Stream:  TESTOOOO 
Status:  Collecting 


(11)  Press  the  Menu  button  to  get  into  PPMS  Main  Menu.  Highlight  collection  to  get 
into  Collect  Data  menu.  Select  stop  and  press  OK. 

(12)  You  will  now  be  at  the  Stop  Options  menu.  Set  a  delay  for  stopping  collection 
using  the  "stop  in  XX  seconds'  option.  Select  ‘stop  at  — ‘to  set  a  specific  time  for  halting 
collection.  Or,  select  stop  now  to  stop  the  collection  immediately.  Then  press  OK. 


^  DIGIT  +  OflLuE  “ 

Xekit 

Stop  Options 

Jk. 

o' 

T 

Stop  in  XX  seoonds 

1 

Stop  at  HH:MM:SS 

[4 

(13)  You  will  now  be  returned  to  the  PPMS  Status  menu.  The  status  will  change  to 

Status:  Ready.  _ 

I  MENU 

PPMS  Status 

Config;  UNTITLED.INI 
Stream;  TESTOOOO 
Status:  Ready 


6.6.5  Sample  Operation  of  the  PPMS  using  the  DCBGUl 
Following  is  an  excerpt  from  the  PPMS  User’s  Guide  that  provides  a  sample  operation 
scenario  for  PPMS  data  collection  using  the  DCBGUl  application.  Operation  requires 
prior  installation  of  the  HPBMS  software  system  (see  the  HPBMS  Overview  document). 
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(Please  note  that  any  IP  address  that  appears  within  this  manual  will  not  work  on  your 
system.  For  appropriate  IP  addresses,  consult  individual  PPMS  or  HPMS 
documentation.) 

(1)  From  the  Start  Menu,  open  the  DCB  GUI  icon.  Follow  the  path  from  the  Start  Menu 
-¥  Programs  — >  Dcae  — >  DCBGUI. 


DCBGUI 


The  DCBGUI  Main  Window  will  then  appear. 
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This  window  will  display  all  the  controls  for  running  the  PPMS:  the  Node  and  Driver 
controls,  the  graphical  node  and  driver  window,  the  Information  area,  and  the  Collection 
controls’,  and  feedback  message  area.  After  startup,  the  only  node  that  will  appear  in  the 
node  view  window  will  be  a  representation  of  the  computer  on  which  the  software  is 
running. 


(2)  The  first  thin£  you  must  do  is  connect  to  a  PPMS.  In  the  System  Controls  area,  press 
Ae  Connect/Close  button. 
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From  the  pull-down  menu  next  to  the  ‘Connect  To  ->’  button,  you  will  need  to  select  the 
IP  address  of  the  PPMS  which  you  want  to  connect  to. 

If  this  is  the  first  time  you  are  using  the  softw^e  or  a  particular  PPMS,  you  will  need  to 
select  the  ‘Add/Delete. . . ’  option  from  the  pull  down  menu.  This  will  bring  you  to  the 
‘Add/Delete  Node  Address’  dialog. 


You  will  need  to  know  the  IP  address  of  the  PPMS  which  you  want  to  connect  to  (see  the 
Front  Panel  Operations  section  of  this  manual).  Enter  that  value  into  fiie  BP 
Address/Name  field,  then  click  ‘Add  >’.  The  IP  address  will  appear  in  the  window  on  the 
right.  If  you  incorrectly  entered  the  address,  select  it  firom  the  right  window,  then  click  ‘< 
Delete’. 

When  you  have  entered  the  correct  address,  you  can  either  click  ‘Done’  to  return  to  the 
Connect  Dialog  and  connect  from  there,  or  select  the  IP  address  from  the  right  window 
and  click  ‘Connect’. 

Note  that  the  PPMS  must  be  fully  powered  up  before  you  v^ll  be  able  to  connect  to  it. 

(3)  After  a  connection  has  been  established,  you  will  be  returned  to  the  DCB  Main 
Window.  Now  2  nodes  will  appear  in  the  node  view  window  -  the  node  representing  the 
computer  that  is  running  the  DCB,  and  the  PPMS. 
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The  -  box  next  to  the  PPMS  icon  indicates  that  there  are  drivers  running  on  the  PPMS, 
listed  below  it.  By  clicking  on  the  -  box,  it  will  ‘close’  the  open  listing  of  the  drivers, 
displaying  only  the  name  of  the  PPMS  and  +  box.  The  +  box  can  be  clicked  at  any  time 
to  re-display  the  drivers. 

(4)  Now  that  you  have  connected  to  the  PPMS  you  are  ready  to  begin  collecting  data. 
From  the  Main  DCB  window,  select  ‘Start’  in  the  Collection  area  of  the  window. 


This  action  will  bring  up  the  start  menu.  From  this  menu  you  can  edit  the  name  of  the 
test  to  be  collected,  and  select  the  start  and  stop  times  for  the  collection.  Start  Now  will 

start  a  collection  as  soon  as  you  press  ‘OK’.  Start  In _ Sec.  will  start  a  collection  in  the 

indicated  number  of  seconds.  This  time  defaults  to  1 0  seconds,  but  you  can  change  this 

number  as  you  see  fit.  The  Start  At _ selection  requires  you  to  indicate  a  time  in  24 

hour  time  when  the  collection  will  begin.  This  is  based  off  of  the  internal  clock  of  the 
DCB,  not  the  internal  clock  of  the  PPMS.  (This  option  is  currently  disabled,  but  will  be 
added  in  later  software  releases.) 
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If  you  are  planning  on  viewing  the  data  in  real  time  using  the  APS,  then  you  must  check 
the  ‘Monitor’  option.  For  more  information  on  the  Monitor  command  and  Monitor 
Interval,  consult  the  later  chapters.  You  can  also  choose  to  monitor  a  collection  after  you 
have  started  collecting  data  by  clicking  the  ‘Monitor’  button  in  the  collection  area  of  the 
Main  DCB  window. 

Checking  ‘Auto  overwrite  existing  files’  will  allow  multiple  collections  with  the  same 
name  to  be  copied  over  each  other. 

Hitting  the  OK  button  will  return  you  to  the  DCB  main  window,  and  messages  will 
appear  in  the  message  window  indicating  that  the  collection  is  waiting  to  start,  and  then 
starting.  In  the  node  view  window  of  the  Main  DCB  window,  you  will  see  that  once 
collection  has  started  the  PPMS  driver  is  listed  as  ‘Collecting’  rather  than  ‘Ready’. 

(5)  When  you  are  ready  to  stop  the  collection,  click  the  ‘Stop’  button  from  the  Collection 
area  of  the  Main  DCB  window. 
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The  Stop  options  are  very  similar  to  the  start  options.  Stop  Now  stops  the  collection  when 

you  click  OK.  Stop  In _ Sec.  stops  the  collection  in  a  number  of  specified  seconds. 

Stop  At _ stops  the  collection  at  a  specific  time.  (This  option  is  currently  disabled, 

but  will  be  added  in  future  software  releases.) 

(6)  After  a  collection  has  ended,  you  can  also  upload  the  data  fi-om  the  Main  DCB 
window,  using  the  upload  button  in  the  collection  area. 


(9)  To  exit  the  DCB  graphical  user  interface,  click  either  the  X  box  in  the  upper  right 
comer,  or  select  the  ‘Exit’  option  firom  the  File  Menu. 

6.6.6  The  PPMS  Configuration  File 

The  PPMS  configuration  file  sets  the  operational  and  data  collection  parameters  of  the 
PPMS  that  are  user  programmable.  This  includes  selection  of  the  programmable  signal 
conditioning  features  of  the  data  collection  channels,  as  well  as  establishment  of  system 
level  parameters  such  as  time,  date,  etc. 

Note:  when  using  the  DCBGUI  or  front  panel  interface  for  operation  of  the  PPMS  the 
user  does  not  need  to  know  the  format  of  the  configuration  file  -  the  interface  provides 
a  user  friendly  means  for  creation  and  editing  of  these  files.  However,  if  the  DCB 
console  is  used  (see  PPMS  User^s  guide  for  details)  the  user  will  need  to  directly  create 
and  edit  the  configuration  files.  Therefore,  a  description  of  the  configuration  file 
format  is  provided  here. 

The  configuration  file  is  stored  on  the  PPMS,  by  default  in  the  c:/dcemmdsl  directory. 
PPMS  configuration  files  can  be  put  elsewhere,  however  the  GUI  looks  only  here  for 
available  configuration  files. 

PPMS  configuration  parameters  are  divided  into  categories  based  on  the  devices  to  which 
they  apply.  Thus,  parameter  names  (or  “keys”)  have  a  multi-level  hierarchical  structure, 
Le.  section. ‘sub-section: key  =  value  .  Some  PPMS  configuration  keys  do 
not  contain  sub-section  entries.  In  this  structure,  section  and  sub-section  (if 
present)  specify  the  PPMS  sub-device  to  which  the  parameter  applies,  key  determines 
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what  aspect  of  that  device’s  configuration  will  be  affected,  and  value  determines  the 
precise  nature  of  the  change. 

In  the  following  discussion,  the  word  “key”  (when  it  does  not  appear  in  mono- spaced 
type)  refers  not  to  the  key  portion  of  the  parameter  name  as  defined  above,  but  rather 
to  the  entire  parameter  name  (i.e.  the  fully-specified  combination  of  sect  ion :  key  or 
sect  ion :  sub-section :  key). 

Conventions:  Sections  and  parameters  are  entered  as  shown,  with  the  following 
exceptions: 

{  n.  .m  }  should  be  replaced  by  a  number  between  n  and  m,  inclusive.  The  braces  {  } 
should  be  omitted.  If  this  notation  appears  within  a  key,  Ihe  number  should  be 
entered  with  no  white  space  between  it  and  the  characters  that  surround  it.  Values, 
however,  can  be  separated  firom  the  assignment  operator  (the  =  sign)  by  one  or  more 
spaces.  In  value  notations,  the  default  value  is  specified  in  boldface.  Example: 
ch{l.  .15}  :amp:gain  =  {  1.  .1000  }  meansthat  chl:amp:gain  = 
500  is  a  valid  key  and  value  combination,  but  that  chi:  amp:  gain  =  1001  or 
chl7:  amp:  gain  =  1  are  not  valid  combinations.  A  gain  setting  of  1  is  indicated 
as  the  default  setting. 

{  cboicel  I  choice2  \  choices  )  expresses  a  list  of  valid  values  fi-om  which 
a  single  value  must  be  chosen.  The  braces  (  /  ) )  and  separators  (  |  )  should  not  be 
included  in  the  value.  The  default  value  of  the  parameter  is  shown  in  bold.  Example: 
sys:  display:  backlight  ={  on  \  off  ;  means  that  the  only  acceptable 
assignments  for  this  key  are  sys:  display:  backlight  =  on  and 
sys:  display:  backlight  =  off,  and  that  the  default  setting  for  the  backlight 
is  “off’. 


See  the  PPMS  User’s  guide  for  more  information  on  configuration. 


6.7  Eye  Tracking  System 
6.7.1  Overview 

The  eye  tracking  system  (ETS)  provides  for  measurement  of  a  user’s  point  of  regard  (gaze 
location)  and  other  eye-related  parameters,  such  as  pupil  size  and  blink.  The  eye  tracking 
hardware  consists  of  a  head  mounted  eye  tracking  device  that  captures  a  video  image  of 
the  eye  as  well  as  a  video  image  of  the  user’s  field  of  view.  The  specialized  software 
system  utilizes  a  client/server  approach,  where  the  user  interface  application  runs  on  a 
client  PC,  and  the  eye  tracking  algorithms  run  on  a  server  PC.  This  two  system  approach 
maximizes  performance  of  the  eye  tracker.  The  eye  tracking  system  may  be  used 
independently  or  as  part  of  the  HPBMS,  where  eye  tracking  results  are  recorded  through 
the  DCAE  software. 
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Figure  47:  Picture  of  the  ETS  Monitoring  Station  (from  left  to  right:  ETS  server  monitor, 
VCR  and  TV  capturing  eye  tracking  results,  VCR  and  TV  capturing  video  of  user 

interface  station) 


Figure  48:  The  Head  Mounted  Eye  Tracking  Device 
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The  ETS  client  application  provides  the  user  interface  and  relays  operational  commands 
to  the  server  application.  It  operates  on  a  Windows  platform  and  has  minimal 
computational  requirements,  since  the  image  processing  algorithms  are  performed  on  the 
ETS  server  PC. 

The  server  application  receives  video  from  the  eye  tracking  device,  controls  the  real-time 
video  acquisition  loop,  processes  the  images,  displays  live  video,  and  handles  the  requests 
of  the  client  by  inserting  tasks  into  the  video  acquisition  loop.  Tasks  include  tracking, 
data  output,  information  updates,  etc.  The  eye  tracking  system  may  be  used  independently 
or  as  part  of  the  HPBMS,  where  eye  tracking  results  are  recorded  through  the  DCAE 
software.  The  server  PC  uses  two  Matrox  Meteor  image  acquisition  cards  for  capture  of 
both  the  eye  image  and  the  field  of  view  image.  These  acquisition  cards  support  frame 
rates  up  to  30Hz  depending  on  frame  size.  With  the  additional  overhead  of  the  image 
processing  algorithms,  the  ETS  outputs  tracking  results  at  a  rate  of  about  24-28  Hz, 
depending  on  the  image  characteristics  (and  the  specifications  of  the  server  PC). 


6.7.2  Image  Processing  Algorithms 

The  eye  tracking  system  uses  a  series  of  algorithms  to  accomplish  the  goal  of  gaze 
determination.  The  algorithms  support  four  major  functions  -  localization,  calibration, 
pupil  detection,  and  automated  focus  of  attention  determination  -  in  order  to  accomplish 
the  goal  of  real-time  eye  gaze  determination  and  cursor  control.  The  following 
subsections  will  describe  these  algorithms  in  detail. 

Localization 


The  localization  process  is  an  optional  feature  (selectable  through  the  client  interface)  that 
can  be  used  to  increase  the  fiume  rate  of  eye  position  determination.  This  process  uses 
image  differencing  to  detect  motion  in  eye  images  and  localize  the  pupil  search  area.  This 
results  in  increased  frame  rates  by  negation  of  false  positives  caused  by  background 
clutter.  When  the  device  is  first  placed  on  the  head,  images  are  captured  as  the  user 
fixates  on  the  four  comers  (extremes)  of  the  viewing  screen.  Images  are  differenced  and 
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then  the  localization  area  (where  eye  motion  is  expected  to  occur)  is  determined  by  the 
region  within  images  where  change  has  occurred  (See  Figure  50). 


Figure  50.  Localization  Window  Derived  from  Binarized  Difference  Image 

Erosion-dilation  algorithms  may  be  used  to  further  refine  the  pupil  search  area,  depending 
on  user  configuration  choices.  Erosion  iteratively  eats  away  at  the  boundaries  of  objects 
in  an  image.  Dilation  then  iteratively  adds  to  the  boundaries  of  objects.  Thus,  erosion  can 
be  used  to  remove  smaller  objects,  and  dilation  can  be  used  to  restore  the  remaining 
objects  in  the  image.  Since  the  movement  of  the  pupil  results  in  a  large  object  around  the 
pupil  area  in  difference  images,  erosion-dilation  will  remove  small  elongated  objects, 
such  as  those  caused  by  eye  lash  movement,  and  retain  the  area  of  pupil  movement  (See 
Figure  50  and  Figure  51) 


Figure  5 1.  Resultant  Erosion-Dilation  Image  Offers  Refined  Localization  Window 

Calibration 

The  eye  tracking  calibration  process  derives  coefficients  for  a  fifth  order  polynomial 
mapping  function  used  to  calculate  gaze  coordinates  from  pupil  coordinates.  The  user 
interactive  portion  of  the  process  requires  the  user  to  focus  on  known  calibration  points 
on  the  screen.  This  results  in  a  set  of  calibration  point  coordinates  and  the  respective 
calculated  set  of  pupil  center  coordinates  derived  during  focus  periods.  Least  squares 
processing  then  determines  the  coefficients  of  a  fifth  order  polynomial  that  optimally 
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maps  pupil  coordinates  to  gaze  coordinates  (See  Figure  52).  The  calibration  process  has 
been  optimized  to  automatically  advance  calibration  points  once  the  user  has  fixated  on 
each  point.  The  default  calibration  process  is  a  9  point  procedure  that  takes  approximately 
30  seconds  to  complete. 


User  looks  at  a  set 
of  predefined  points 


(^^y)  pupil 


Qaast  Squares  fit  ovi 
(x,y)HMD  =  5th  order  fun 


ir  all  points: 
-jtion  of  (x,y)  pupil 


Figure  52:  Interactive  Calibration  Process 


Punil  Detection  and  Center  Determination 

The  pupil  detection  algorithms  begin  with  identification  of  a  seed  point  within  the  pupil 
area.  This  can  be  performed  by  one  of  two  methods,  which  may  be  selected  through  the 
ETS  client  application:  1)  using  a  threshold  comparison,  or  2)  using  the  reflection  (glint) 
generated  on  the  eye  from  the  illumination  LED.  The  default  method  is  a  threshold 
comparison,  because  it  is  the  most  effective  for  the  range  of  possible  users.  The  glint 
based  approach  can  be  more  efficient,  but  should  be  used  with  caution  because  for  some 
user’ s  the  glint  does  not  always  remain  within  the  pupil. 

Once  found,  the  seed  pixel  is  then  utilized  for  an  intensity/density  based  region  growing 
algorithm.  The  region  growing  algorithm  fills  the  pupil  area  until  certain  conditions  are 
met  that  are  based  on  the  density  of  like  neighborhood  pixels  and  the  intensity  of  the  pixel 
as  compared  to  the  seed  pixel.  The  two  extreme  horizontal  points  of  the  pupil  are  then 
used  to  determine  the  center  of  the  pupil.  Use  of  the  horizontal  pupil  extremes  allows  for 
accurate  determination  of  the  pupil  center  even  when  there  is  significant  pupil  occlusion 
caused  by  the  eyelid. 

As  a  final  step,  the  pupil  center  is  then  mapped  to  a  gaze  coordinate  (within  the  field  of 
view  image)  using  the  mapping  fimction  derived  from  the  calibration  process.  Other 
parameters  that  are  generated  from  the  pupil  detection  algorithm  are  the  pupil  size  and 
blink  detection. 
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Automated  Focus  of  Attention 

Prior  to  development  of  the  focus  of  attention  algorithms  evaluation  of  a  user’s  point  of 
regard  had  to  be  performed  by  a  observer  watching  the  recorded  video  and  making  note  of 
where  the  user  is  looking.  This  is  because  as  the  user  moves  around  the  objects  in  the 
captured  field  of  view  also  move  around.  However,  the  developed  focus  of  attention 
algorithms  provide  for  quantitative  determination  of  gaze  location  relative  to  a  “tagged” 
region  within  the  user’s  field  of  view  (such  as  a  computer  monitor). 

Quantitative  determination  of  the  user’s  point  of  regard  is  of  significant  advantage 
because  it  allows  for  automated  analysis  and  numerical  manipulation  of  a  user’s  gaze 
location.  For  instance,  one  can  easily  perform  calculations  of  how  often,  how  many  times, 
or  how  long  a  user  looks  at  a  certain  object  or  area. 

The  basic  algorithmic  process  is  based  on  determination  of  the  center  of  the  monitor  (or 
tagged  region).  This  is  performed  by  tagging  the  monitor  with  identifiable  markers,  as 
shown  in  Figure  55.  The  monitor  tags  used  are  four  2-inch  diameter  red  circles  used  to 
mark  out  a  rectangular  area  in  which  the  software  will  compute  X  and  Y  coordinates. 

They  were  designed  to  be  placed  on  the  fi-ame  of  a  monitor,  to  calculate  the  gaze  position 
on  the  screen,  or,  they  can  be  placed  on  a  wall,  floor,  ceiling,  vehicle  windshield  or  any 
flat  surface  to  provide  a  coordinate  system. 

Monitor  Tags  must  be  placed  in  a  diamond  formation,  with  the  intersection  of  the 
horizontal  pair  and  the  vertical  pair  being  at  the  center  of  the  area  in  which  you  wish  to 
determine  a  coordinate  system. 
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The  Monitor  Tags  form  a  coordinate  system  based  on  the  following  diagram.  Results 
outside  of  the  monitor  boundary  (i.e.  X,Y  <  0,0  or  X,Y  >  Xmax,  Ymax)  are  valid. 


X^naXfY^nax 


Accurate  determination  of  the  center  point  can  be  determined  while  only  3  of  the  tags  are 
within  the  captured  field  of  view  image,  allowing  for  signification  motion  of  the  user. 
Based  on  this  determination,  the  gaze  location  computed  by  the  eye  tracking  algorithms  is 
then  converted  to  a  specific  coordinate  location  within  the  monitor  screen.  This 
computation  takes  into  account  the  particular  screen  size  and  accounts  for  the  rotation 
angle. 
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6.7.3  Sample  (Quick  Start)  Operation  of  the  Eye  Tracking  System 
This  section  contains  an  example  of  only  the  basic  procedines  necessary  to  start  and  stop 
a  single  session  of  eye  tracking.  For  more  detailed  information  refer  to  the  Eye  Tracking 
System  User’s  Guide.  This  procedure  requires  that  the  Human  Performance  Based 
Measurement  System  Software  be  previously  installed. 

6.7.3.1  Starting  the  ET  Server 

Once  the  ET  Server  has  started,  you  will  be  presented  with  a  login  prompt.  See  the  ETS 
User’s  Guide  for  login  information. 

After  you  have  successfully  logged  in,  at  the  command  prompt  type:  “startx" 

This  will  start  the  X-Windows  interface  firom  which  the  ET  Server  is  run. 

All  operations  from  here  on  are  performed  in  the  ‘xterm’  window  on  the  desktop. 

Change  directories  by  typing  the  following;  “cd  /EYETRACKER/bin”.  Then,  type  in 
“./EyeTrackerServer”  to  begin  the  server  application.  You  will  see  the  following: 


6.7.3.2  Starting  the  Calibration  Client 

This  step  is  only  necessary  if  you  are  going  to  be  running  eye  tracking  experiments  on  a 
different  computer  than  the  one  running  the  ET  Client.  The  Calibration  Client  Icon  can 
be  reached  through  the  Start  Menu  Programs  DCAE  EyeTracker  Calibration 
Client  path  on  the  subject  interface  machine.  The  program  will  run  minimized,  so  as  not 
to  take  up  and  space  on  the  screen. 

6.7.3.3  Starting  the  ET  Client 

On  the  ET  Client  computer,  select  the  EyeTracker  icon  from  the  Start  Menu  Programs 
DCAE  EyeTracker. 


EyeT  racker 


The  main  window  of  the  interface  will  appear,  and  the  message  console  window  will 
appear  minimized.  At  startup  the  ET  Client  should  automatically  connect  to  the  ET 
Server.  Connection  may  not  occur  if  the  Server  IP  address  is  not  set  properly  or  other 
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networking  problems  exist.  In  this  case,  fix  the  problem,  and  then  select  the 
“Coimections”  menu  in  the  Client  interface,  and  choose  “connect  to  the  ET  Server.” 


On  the  ET  Server  machine,  the  following  will  appear  in  die  text  window,  and  2  video 
feed  windows,  “External”  and  “EyeTrackerServer,”  will  appear.  The  two  video  windows 
may  appear  overlapping  each  other,  and  you  will  want  to  move  them  so  that  you  have  an 
unobstructed  view  of  each. 


6.7.3.4  Wearing  the  Eye  Tracker  Correctly 

The  Eye  Tracker  should  be  placed  on  the  head  so  that  the  arms  sit  above  the  ears,  much 
like  a  pair  of  glasses,  and  the  brim  should  stick  out  in  front  like  the  rim  of  a  baseball  cap. 
The  subject’s  left  eye  should  be  situated  directly  in  front  of  the  glass  mirror.  The  mirror 
should  not  be  resting  against  the  subject’s  cheek.  The  Eye  Tracker  is  designed  to  fit 
tightly,  to  avoid  motion  artifacts,  but  not  painfully.  If  the  Eye  Tracker  bands  are  too  tight, 
they  can  be  adjusted  slightly  with  a  flathead  screwdriver. 
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The  Eye  Tracker  must  be  placed  so  that  the  subject’s  pupil  is  completely  visible  in  the 
EyetrackerServer  window.  This  can  be  acheived  by  manipulathig  the  placement  of  the 
Eye  Tracker  on  the  subject’s  head. 


If  you  are  using  Monitor  Tags,  then  use  the  External  window  to  make  certain  that  at  least 
3  of  these  tags  are  visible  in  the  External  video  feed.  You  may  need  to  move  the  position 
of  the  subject,  or  adjust  the  angle  of  the  external  camera. 


6.7.3.5  The  Calibration  Procedure 

Once  the  ET  Server  is  running,  and  the  Eye  Tracker  is  arranged  on  the  subject’s  head  and 
they  are  positioned  in  front  of  a  wall-moimted  calibration  grid  or  a  monitor  running  a 
Calibration  Client,  click  the  Calibration  button.  The  subject  will  then  focus  on  each  point 
on  the  grid  either  as  they  appear  (monitor  calibration)  or  as  indicated  by  an  audio  tone 
(wall  grid  calibration). 
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The  points  should  be  followed  in  a  top-to-bottom,  left-to-right  manner. 

You  can  check  calibration  by  watching  the  External  view  on  the  ET  Server.  ■\^en  the 
software  has  calibrated  a  point,  you  will  see  a  red  circle  and  crosshairs  appear  in  the 
External  view  at  the  location  the  software  thinks  the  subject  is  looking. 

If  calibration  is  running  slowly,  and/or  you  think  there  is  an  error  with  the  system,  you 
can  cancel  calibration  by  clicking  the  mouse  into  the  calibration  window  (on  the  system 
that  is  running  the  ET  Client)  and  pressing  CTRL-X  on  the  keyboard.  The  most  common 
problem  is  that  the  software  is  not  finding  the  pupil.  While  the  ET  Server  is  active,  and  a 
subject  is  wearing  the  eye  tracker,  you  should  see  a  red  box  surrounding  the  pupil  as  an 
indicator  that  the  software  is  finding  the  pupil.  This  box  (and  sometimes  a  small 
crosshairs  inside  the  box)  will  blink  on  and  off.  The  red  box  should  be  on  more  than  it  is 
off  If  you  do  not  see  tiie  red  box,  it  means  that  the  software  is  not  finding  the  pupil,  and 
therefore  cannot  track  the  eye. 


Another  means  of  determining  if  you  are  finding  the  pupil  is  by  watching  the  output  in  the 
xterm  window  on  the  ET  Sever  machine.  When  the  algorithm  is  finding  the  pupil,  a 
listing  of  the  fiume  rates  will  scroll  down  the  window. 


Frame  Rate:  9*841125 

Frame  Rate:  10*684853  i  t  *  a 

Pupil  search  fail^viiScore;  0#TO8315  Thresh:  0*780000 

Frame  Rate:  10*6S8404 

Frame  Rate:  10 ♦886005 

Frame  Rate:  10*381205 

Frame  Rate:  10*641735 

Frame  Rate:  10*661515 

Frame  Rate:  iO*9iG995 

Frame  Rate:  10*677998 

Frame  Rate:  10*496965  J 

Pupil  search  failed* :♦  ♦Score:  0*758865  Thresh:  0*760000 
Frame  Rate: 
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When  the  algorithm  is  not  finding  the  pupil,  the  window  will  display  that  the  pupil  search 
failed. 


search  failed* ♦♦Score} 
Pupil  search  failed* ♦♦Score} 
Frame  Rate}  5*493459 
Pupil  search  failed* ♦♦Score} 
Pupil  search  failed* ♦♦Scdre} 
Pupil  search  fai led* ♦ ♦Score; 
Pupil  search  fai led* ♦♦Score} 
Pupil  search  failed** ♦Score; 
Pupil  search  fai led* ♦♦Score} 
Pupil  search  failed* **Score} 
Frame  Rate;  5*601264 
Pupil  search  fai led; ♦♦Score; 
Pupil  search  failed**^*Score} 


0»715727  thresh:  0.760000 
0*742746  Thresh:  0,^0000 


0i759171 

0.724644 

0*749225 

0*744173 

0^723196 

02670728 

0*719467 


Thrtsh: 

Thresh: 

thresh: 

thresh: 

Thresh: 

thresh: 

Thresh: 


0*760000 

0*760000 

0*760000 

0*760000 

0*760000 

0*760000 

0,760000 


0:752263  thresh:  0*760000 
0*759154  Thresh:  0*760000 


Generally,  no  matter  what  settings  you  are  using,  the  algorithms  will  occasionally  fail  to 
find  the  pupil.  A  good  tracking  session  will  find  the  pupil  more  often  than  it  does  not.  If 
you  are  having  trouble  calibrating  a  subject,  you  may  need  to  adjust  the  settings. 

6.7.3.6  Tracking  the  Eye 

After  calibration,  all  red  indicators  in  the  External  and  EyeTrackerServer  windows  will 
disappear.  By  clicking  the  Track  button,  the  software  will  begin  to  track  gaze  position, 
and  that  will  be  indicated  in  the  External  view  by  a  red  crosshairs.  The  pupil  should  also 
again  be  surrounded  by  a  red  box  in  the  EyeTrackerServer  window. 


6.7.3.7  Running  A  Collection  through  the  DCB 

Collecting  Eye  Tracking  data  through  the  DCB  is  done  as  a  standard  DCB  collection  (for 
more  information  on  the  DCB  and  its  operation,  please  refer  to  the  DCB  Manual). 

First,  open  the  DCB  by  double  clicking  on  it’s  icon.  (Now  you  will  have  both  the  ETS 
and  the  DCB  running.) 
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Before  the  DCB  will  accept  eye  tracker  data,  you  must  load  the  EyeTracker  driver. 

Choose  the  ‘Load  Drivers’  button,  then  from  within  the  ‘Load  Drivers  Dialog’,  load  die 
eye  tracker  driver  for  the  node  on  which  you  are  running  the  ET  Client. 

Start  a  DCB  collection.  You  may  start  a  collection  at  any  time  during  the  first  6  steps 
(from  Starting  the  ET  Sever  3.1  through  Tracking  the  Eye  3.6).  However,  data  is  not 
collected  into  a  DCB  stream  until  the  Eye  Tracker  has  actually  begun  tracking  the  eye 
(3.6).  Once  the  Track  button  in  the  ET  Client  has  been  clicked,  the  Eye  Tracker  will  send 
data  to  the  DCB,  even  after  a  DCB  collection  has  ended.  You  must  stop  tracking  using 
the  ET  Client  in  order  to  stop  data  flowing  into  the  DCB. 

6.7.3.8  Stopping  an  Eye  Tracking  Session 

To  stop  tracking  the  eye,  simply  click  on  the  Stop  Eyetracking  button.  (Note  that  this  will 
only  stop  the  operation  of  the  Eye  Tracker;  it  will  not  stop  a  DCB  collection,  even  a 
collection  tiiat  is  running  the  EyeTrackerDriver.) 

6.7.3.9  Exiting  the  Eye  Tracking  System 

Before  exiting  the  ET  Client,  disconnect  from  the  ET  Server  through  the  Connections 
Menu.  This  will  also  shut  down  the  ET  Server. 

You  should  also  disconnect  from  the  Calibration  Client  at  this  time,  if  you  were  running 
one. 

To  exit  the  Eye  Tracker,  simply  click  the  comer  X  button  of  the  main  ET  Client  window, 
or  select  the  Exit  option  from  Ae  Action  Menu. 
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6.7.4  Eye  Tracking  System  Configuration  and  Control  Options 
Calibration  Options 

The  following  options  can  be  modified  tiirough  the  client  interface. 

1.  Save  calibration  results  on/off. 

2.  Screen  Size  -  this  is  for  determining  the  size  of  the  screen  where  the  computerized 
calibration  grid  will  be  displayed. 

3.  Remote  Calibration  -  checking  this  box  will  display  the  calibration  grid  on  a  remote 
machine.  That  remote  machine  must  be  running  the  Calibration  Client,  and  the  ET 
Client  must  be  connected  to  it  (see  the  Connections  Menu). 

4.  Calibration  Type  Selection: 

•  Wall  Grid  Calibration  -  this  is  for  using  a  wall  grid  calibration 

•  Monitor  Based  Calibration  -  this  is  used  to  select  the  computer  to  display  a 
calibration  grid  on  a  screen.  It  is  used  in  conjunction  with  Remote  Calibration 
and  Screen  Size  options. 

Vision  Aleorithnts  Ovtions 

The  following  options  can  be  modified  through  the  client  interface: 

1.  Display  Processed  Video  on/off. 

Choosing  display  processed  video  will  inform  the  server  to  display  live  video  of  the 
users  eye  on  the  server’s  monitor.  The  video  will  be  marked  with  a  square  to  show 
pupil  diameter,  a  large  cross  to  show  pupil  center,  and  a  small  cross  to  show  seed 
location. 

For  optimum  performance  processed  video  should  not  be  displayed  as  this  slows  the 
system  down  considerably. 

However,  having  the  processed  video  displayed  can  often  be  of  great  assistance  in 
centering  a  subject’s  eye  in  the  camera  view  (for  better  performance),  and  can  be  used 
to  ‘debug’  why  certain  subjects  have  poor  performance. 

2.  Display  Secondary  Video  on/off. 

Choosing  display  secondary  video  will  inform  the  server  to  display  the  video  from  the 
second  camera  (the  external  world  view).  The  video  will  be  marked  with  a  single  red 
cross  to  indicate  gaze  location.  It  may  also  be  marked  with  a  small  red  box  indicating 
the  center  of  the  Monitor  Tags  if  they  are  being  used. 

Since  the  information  gathered  from  the  eye  tracker  is  mostly  visual,  display 
secondary  video  should  be  on.  However,  it  can  be  turned  off  to  increase  the  update 
rates  if  the  results  are  to  be  viewed  in  another  form  (such  as  the  raw  data  stored  in  the 
output  file,  or  collected  through  the  DCB  software). 

3.  Find  Monitor  Tags 

This  will  indicate  that  the  software  should  look  for  4  red  dots  arranged  in  the  staridard 
positions.  This  is  used  to  give  a  set  of  coordinates  when  the  subject  is  looking  within 
the  area  of  the  tags. 
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4.  Mark  Monitor  Center 

This  is  useful  only  if  you  are  displaying  the  secondary  video,  and  using  Monitor  Tags 
and  the  Find  Monitor  Tags  box  is  checked.  It  will  place  a  small  square  at  the 
intersection  of  the  lines  created  by  the  Monitor  Tags,  and  appear  in  the  Secondary 
Video  window. 

5.  Glint  Based  Region  Growing  /  Dark  Pupil  Based  Region  Growing  radio  buttons 
The  region  growing  routine  can  determine  the  pixels  which  makeup  the  pupil  when 
supplied  with  one  pixel  within  the  pupil.  There  are  two  methods  to  finding  a  point 
inside  the  pupil: 

•  Glint  Based  Region  Growing  uses  the  IR  reflection  (glint)  as  a  guide  to  pupil 
location.  It  is  assumed  that  the  pupil  vwll  be  within  a  certain  area  above  the  glint. 
Therefore,  a  vertical  offset  is  added  to  the  glint  location  and  the  result  is  used  as 
the  seed  location. 

•  Dark  Pupil  Based  Region  Growing  uses  the  dark  characteristics  of  the  pupil  to 
find  a  point  within  the  pupil.  Basically,  the  dark  areas  of  the  image  are  analyzed 
to  determine  where  the  pupil  is  most  likely  to  be.  Then  this  location  is  used  as  the 
seed  to  the  region  growing  algorithm  to  produce  a  more  accurate  determination  of 
the  pupil  center.  This  is  the  default  setting  because  it  generally  produces  the  best 
results  with  the  eyetracker  supplied. 

Servers  Options 

The  following  parameters  can  be  modified  through  the  client  interface: 

1 .  Eye  Tracker  Server  IP  Address 

2.  Eye  Tracker  Client  Input  and  Output  Sockets 
Input  Socket  defaults  to  3508. 

Output  Socket  defaults  to  3507. 

Both  of  these  values  must  be  the  same  as  the  Eye  Tracker  Server,  and  are  by  default. 

If  they  need  to  be  changed,  refer  to  the  Eye  Tracker  Server  Section  in  this  manual. 
Whenever  socket  numbers  are  changed,  the  client  must  be  restarted  before  they  will 
take  effect. 

3.  Calibration  Client  IP  Address 

4.  Calibration  Client  Input  and  Output  Sockets 
Input  Socket  defaults  to  3516. 

Output  Socket  defaults  to  3515. 

Both  of  these  values  must  be  the  same  as  the  Calibration  Client,  and  are  by  default.  If 
they  need  to  be  changed,  refer  to  the  Calibration  Client  section  in  this  manual. 
Whenever  socket  numbers  are  changed,  the  client  must  be  restarted  before  they  will 
take  effect. 

Files  Options 

The  following  parameters  can  be  modified  through  the  client  interface: 

1.  Calibration  File 
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Used  to  create  calibration  grid  for  computer  screen  calibrations.  This  file  is  nothing 
more  than  a  list  of  X  and  Y  coordinates,  1  coordinate  set  per  line,  that  indicate  where 
on  the  screen  points  should  be  placed.  Points  are  used  for  calibration  one  at  a  time, 
from  the  top  of  the  file  to  the  bottom: 

100  100 

200  200 

300  300 


Threshold  Controls 


Figure  54:  The  Threshold  Control  Interface 


The  Threshold  window  is  used  to  adjust  the  mathematics  of  the  algorithms  used  to  detect 
and  track  the  pupil.  Difficulties  that  occur  because  the  software  is  not  ‘finding’  the  pupil 
can  be  corrected  here. 

The  following  parameters  can  be  modified  under  the  Thresholds  Option: 

1 .  Dark  Pupil  Intensity  Threshold 

This  value  indicates  the  brightness  or  darkness  that  the  software  expects  the  pupil  to 
be.  The  lower  the  value,  the  software  looks  for  a  darker  pupil;  the  higher  the  value, 
the  software  looks  for  a  brighter  pupil.  This  is  examined  on  a  pixel  by  pixel  basis 
across  the  entire  image  captured  by  the  Matrox  cards. 

2.  Region  Growth  Intensity  Threshold 

The  range  that  a  nearby  pixel  must  be  within  in  order  to  be  considered  a  part  of  the 
dark  pupil.  This  value  is  used  to  look  for  the  edges  of  the  dark  pupil.  This  variable 
should  generally  be  set  low  to  avoid  the  software  blending  the  pupil  and  the 
surroimding  white  eye  togther. 

3.  Region  Growth  Density  Threshold 

The  number  of  neighboring  pixels  that  must  meet  the  above  two  criteria  in  order  for 
the  initial  pixel  to  be  considered  part  of  the  pupil. 

4.  Pupil  Score  Threshold  (%) 

The  Pupil  Score  is  a  measure  of  how  well  the  shape  of  the  detected  pupil  matches  the 
shape  of  the  ideal  pupil.  Requirements  for  a  perfect  match  would  be  100%,  but  this  is 
unrealistic.  Generally,  a  70%  match  (the  default  setting)  produces  good  performance. 
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Output  Format: 

The  Output  Format  Window  is  used  to  determine  which  bits  of  data  are  included  in  any 
data  files  created  or  exported  to  the  DCB  software.  When  running  the  eyetracker  through 
the  DCAE,  all  fields  must  be  selected  to  be  a  part  of  the  Output  Data. 

The  following  parameters  can  be  modified: 

1.  Text  File  on/off 

Toggles  the  creation  of  a  text  file  (containing  the  data  indicated  below)  named 
eyetrack.dat.  The  created  file  will  appear  in  the  directory  that  the  ET  Client  is  in. 

2.  Data  Socket 

This  is  the  value  of  the  data  socket  connection  to  the  DCAE.  It  must  match  the  socket 
number  listed  in  the  eyetracker.ini  file.  For  more  information,  consult  the  DCAE 
Software  manuals  (specifically  the  DCB  User’s  Guide). 

3.  Pupil  Center  on/off 

Toggles  the  inclusion  of  the  X  and  Y  pupil  center  data 

4.  Gaze  Position  on/off 

Toggles  the  inclusion  of  the  X  and  Y  gaze  position  data 

5.  Time  Stamp  on/off 

Toggles  the  inclusion  of  the  ET  Server  time  stamp  data.  (The  DCB  time  stamp  will 
have  some  transfer  delays,  so  the  ET  Server  time  stamp  can  be  used  to  provide  the 
most  accurate  timing  information,  when  desired). 

6.  Monitor  Gaze  Position  on/off 

Toggles  the  inclusion  of  the  gaze  position  data  which  has  been  set  to  a  coordinate 
system  based  off  of  the  Monitor  Tags.  If  Monitor  Tags  are  not  going  to  be  used 
during  a  collection,  this  data  field  will  return  incorrect  data. 

7.  Pupil  Size  on/off 

Toggles  the  inclusion  of  Pupil  Size  data. 

8.  Pupil  Score  on/off 

Toggles  the  inclusion  of  the  Pupil  Score  data.  The  Pupil  Score  is  a  rating  of  how  well 
the  software  tbink.s  it  captured  and  tracked  the  pupil  during  a  particular  firame. 

9.  Blink  Status  on/off 

Toggles  the  inclusion  of  the  Blink  Status  data.  Blink  Status  is  a  field  indicating  if  the 
eye  was  blinking  during  a  particular  frame. 


6.8  Body  Tracking  System  Interface 
6.8.1  Overview 

The  Firefly  optical  motion  capture  system  is  designed  to  track  the  motion  of  any  object  (a 
human  body,  vehicles,  models,  etc. . .).  It  is  ideally  suited  for  the  development  of 
computer  games,  the  production  of  computer  animation,  as  a  input  device  for  virtual 
reality  applications,  and  in  providing  data  for  human  biomechanic  or  ergonomic  studies. 
The  most  common  systems  used  today  utilize  either  magnetic  field  or  optical  tracking 
techniques  to  capture  human  motions.  In  comparison  with  the  Firefly,  the  magnetic 
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systems  are  costly,  not  as  accurate  and  suffer  from  inherent  limitations  when  used  near 
metal  objects  and  structures.  The  optical  systems  most  commonly  used  today  are 
accurate,  very  expensive,  and  require  extensive  calibration  before  the  motion  capture 
sessions,  and  off-line  processing  after  the  sessions. 

The  basic  Firefly  motion  capture  system  consists  of  one  camera  array,  one  tag  controller, 
32  tags,  and  accompanying  cables.  The  Firefly  measurement  system  consiste  of  three 
charge-coupled  device  (CCD)  cameras  mounted  on  a  1  meter  rigid  bar.  By  unaging 
infrared  light  emitting  diode  (IR  LED)  tags,  this  camera  array  can  accurately  determine 
their  3D  position  in  real  time.  Since  the  camera  array  is  precalibrated,  there  is  no 
calibration  required  which  means  that  you  can  immediately  begin  using  the  system  to 
accurately  track  tags. 

The  tag  controller  is  a  small,  battery  operated  waist  mounted  unit  that  is  used  to  strobe  the 
active  tags.  The  tag  controller  does  not  need  to  be  tethered  to  the  array,  since  the  system 
is  auto  synchronizing  when  the  tags  are  in  view  of  the  array.  Should  the  tags  be  not  in 
near  continuous  view  of  the  array,  then  a  synchronizing  cable  is  recommended  (included.) 
The  system  can  be  configured  to  track  up  to  256  tags  with  multiple  slaved  tag  controllers. 

The  tags  are  small,  wide  angle  emission  IR  LEDs  that  are  virtually  massless.  They  are 
designed  to  be  attached  by  adhesives  to  a  persons  skin,  or  can  be  attached  via  thread  to 
gloves,  elastic  bands,  special  garments,  etc.  To  be  tracked  they  must  be  in  the 
simultaneous  field  of  view  of  all  three  cameras  in  the  camera  array.  When  not  in  view, 
the  data  can  be  filtered  to  produce  continuous  uninterrupted  data  at  a  greater  latency. 
Alternately,  multiple  camera  arrays  can  be  slaved  together  to  eliminate  virtually  all  line  of 
sight  problems. 
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Figure  55:  Illustration  of  the  Firefly  System  Operation 


This  system  was  developed  through  other  research  and  development  efforts,  but  was 
identified  for  integration  into  the  HPBMS  early  in  this  Phase  II  effort.  As  a  result,  the 
interface  capabilities  and  data  viewing  tools  were  integrated  into  the  HPBMS.  While  the 
Firefly  system  itself  is  not  a  part  of  the  delivered  system  for  this  Phase  H,  it  can  be  readily 
integrated  into  the  system  if  acquired  separately. 


6.8.2  Sample  Operation  of  Body  Tracker 

Following  is  an  excerpt  from  the  Body  Tracker  User’s  Guide  that  provide  a  “Quick  Start 

guide  to  operation  of  the  Firefly  within  the  HPBMS.  Refer  to  the  User’s  Guide  for  more 

details. 

1 .  Start  the  DCB  GUI.  It  can  either  be  reached  from  it’s  icon  in  the  DCAE  folder,  or 
from  the  Start  Menu,  Programs,  Dcae,  DCB  GUI. 

2.  Load  the  Bodytracker  driver.  Under  the  Drivers  option  from  the  main  GUI  window, 
select  the  ‘Load’  button.  From  the  Node  pull-down  menu,  select  the  machine  that  the 
Firefly  is  connected  to.  From  the  Driver  pull-down  menu,  select  Body  Tracker.  You 
may  give  the  driver  an  alias.  If  a  configuration  file  exists  for  the  Body  Tracker,  it  can 
be  edited  by  clicking  change  at  this  point.  If  a  configuration  file  does  not  yet  exist, 
the  driver  must  be  loaded  with  the  default  settings.  Add  the  driver  by  clicking  Add, 
then  click  Done.  The  driver  is  now  loaded. 

3 .  Start  a  collection.  The  Firefly  must  be  turned  on,  connected  to  the  computer  running 
the  DCB  GUI,  and  foe  Tag  Controller  must  be  turned  on. 
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4.  Open  the  APS  (if  necessary).  If  the  APS  is  already  open,  refresh  the  stream  view  so 
that  the  new  data  being  collected  will  be  available  to  view. 

5.  Select  all  of  the  necessary  data. 

6.  Open  the  Bodytracker  Viewer. 

7.  If  you  are  viewing  data  in  real-time,  from  the  Viewer  Menu,  select  the  ‘Real-Time’ 
option.  If  you  are  playing  back  previously  recorded  data,  open  the  Playback  Controls 
from  the  Playback  Menu.  Note  that  the  Bodytracker  Viewer  works  best  when 
displaying  data  by  Frame  #.  This  setting  can  be  changed  through  the  Playback 
controls. 

8.  Create  and/or  Select  a  configuration  file.  A  configuration  file  will  draw  lines  to 
connect  the  points  of  data  being  displayed  to  better  represent  the  object  being  tracked. 
For  more  detailed  information  on  how  to  create  a  configuration  file,  see  below.  From 
the  Viewer  Menu,  select  ‘Properties’,  then  the  ‘Configure’  option.  Select  the 
appropriate  configuration  file,  and  it  will  automatically  be  applied  to  the  data  being 
displayed. 

9.  Exit  the  viewer  when  finished. 
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7.  Experimental  Procedures  and  Results 


7.1  Driving  Simulation  Experiment 

As  a  preliminaiy  evaluation  and  assessment  of  the  HPBMS  concept,  we  conducted 
driving  simulator  (computer  interface)  experiments,  during  the  early  stages  of  the  Phase  11 
system  integration  process.  This  effort  was  performed  in  conjunction  with  an  Air  Force 
SBIR  project  that  was  focused  on  the  development  of  innovative  cognitive  capacity 
measurement  system.  This  preliminary  evaluation  demonstrated  the  feasibility  and 
potential  benefit  of  EEG  and  other  measures  for  cognitive  and  other  performance 
assessment. 


7.1.1  Experiment  Description 

The  driving  task  experiment  was  conducted  assess  the  feasibility  and  potential  benefit  of 
physiological  measurement  for  cognitive  and  other  performance  assessment.  The  specific 
protocol  was  designed  to  begin  studying  the  neuro-cognitive  and  psychophysiological 
responses  to  differing  levels  of  workload  in  a  visuo-motor  task.  Driving  was  performed 
using  computer-based  driving  simulation  tools. 

The  driving  simulator  has  several  variables  that  can  be  altered  such  as  track  geometry, 
lane  indicators,  visibility,  frame  rate,  etc.  For  this  initial  experiment  only  one  parameter  - 
track  geometry  -  was  varied,  as  a  means  to  creating  different  cognitive  load  requirements. 
Full  force  feedback  on  the  steering  wheel  and  auditory  feedback  were  provided,  with  the 
graphics  displayed  on  the  computer  monitor  at  the  maximum  frame  rate  (25  frames  per 
second)  and  a  zero  frame  lag  setting. 

Electrodes  and  sensors  were  placed  on  the  subject  to  measure  the  signals  listed  in  Figure 
56.  A  limited  set  of  measurements  was  used  for  this  initial  experiment. 


Figure  56:  Physiological  Signals  Measured  for  Preliminary  Driving  Simulator 
_  Experiment _ _ _ 
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The  experiment  consisted  of  an  8  minute  session,  as  shown  in  Figure  57.  Subjects  were 
seated  at  the  driving  simulation  station,  consisting  of  the  computer  screen,  steering  wheel, 
pedals,  and  keyboard.  First  the  subject  was  asked  to  rest  for  1  minute  with  eyes  open,  and 
then  rest  for  an  additional  minute  with  eyes  closed,  in  order  to  obtain  baseline 
measurements.  Then  two  driving  tasks  were  performed.  Each  task,  including  program 
startup  time,  lasted  approximately  three  minutes.  The  order  in  which  the  subject 
performed  the  tasks  (driving  on  easy  track  and  driving  on  difficult  track)  was  randomly 
selected. 


Figure  57:  Phase  I  Experiment  Stages 


State  /  Task 

Duration 

Resting  -  eyes  open 

1  minute 

Resting  -  eyes  closed 

1  minute  ■ 

Driving  -  track  1 

3  minutes 

Driving  -  track  2 

3  minutes 

In  both  task  conditions,  the  subject  was  asked  to  drive  as  fast  as  they  could  without 
leaving  the  road.  It  was  stressed  that  both  driving  speed  and  remaining  on  the  track  were 
equally  important.  The  subject  controlled  the  car  by  using  a  keyboard  to  start  the  car  and 
to  change  gears  of  the  car.  Steering  wheel  and  pedals  (gas  and  brake)  were  used  to 
control  direction  and  speed.  A  picture  of  the  driving  simulation  display  interface  is  shown 
in  Figure  58. 
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Figure  58:  Picture  of  Driving  Simulator  Display 

Complete  data  collection  during  the  experiment  included  the  physiological  signals  that 
were  listed  in  Figure  56,  video  and  audio  captured  using  the  VCR,  and  the  following 
driving  parameters  captured  from  the  simulator: 

Position  on  the  track 
Speed 

Steering  wheel  position 

Gear 

RPM 

Brake  pedal  activation 
Gas  pedal  activation 
Soxmd  (on/off) 

Frame  rate 
Frame  lag 
Off  road  (left/right) 

Force-feedback  activations 
Collisions 
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Figure  59:  Picture  of  Laboratory  Setup  for  Driving  Simulation  Experiments 


7.1.2  Results  and  Discussion 

Several  validation  exercises  and  two  complete  were  performed  as  part  of  this  effort  for 
preliminary  assessment  of  the  HPBMS  concept.  Robust  statistical  analysis  of  the  data  is 
not  possible  without  a  much  larger  set  of  experiments  (to  be  conducted  through  future 
research  effort).  As  such,  the  following  figures  and  data  present  the  results  of  one  of  the 
experiments  as  a  demonstration  the  potential  effectiveness  of  the  developed  system,  and 
to  suggest  the  benefit  of  continued  development,  testing,  and  research. 

Figure  60,  Figure  61,  and  Figure  62  show  measured  EEG  during  eyes  open  versus  eyes 
closed  at  three  different  electrode  locations  (T6,  P4,  and  C4).  These  plots  demonstrate 
strong  alpha  waves  during  eyes  closed  (particularly  at  location  T6)  and  alpha  suppression 

during  eyes  open. 
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Frequency  (Hz)  Frequency  (Hz) 


Figure  60:  EEG  Characteristics,  at  T6-A2,for  Eyes  Open  Versus  Eyes  Closed, 
a)  3  second  segment  of  EEG  during  eyes  open  period,  b)  3  second  segment  of  EEG 
during  Eyes  Closed  period,  c)  EEG  spectrum  over  entire  60  seconds  of  Eyes  Open  period, 
d)  EEG  spectrum  over  entire  60  seconds  of  Eyes  Closed  Period.  Spectrum  plots  indicate 
strong  alpha  wave  content  during  eyes  closed  and  alpha  suppression  during  eyes  open. 


129 


CYBERNET  SYSTEMS  CORPORATION 


a)  EEG  (P4)  -  Eyes  Open  b)  EEC  (P4)  -  Eyes  Closed 


Figure  61:  EEG  Characteristics,  at  P4-A2,  for  Eyes  Open  Versus  Eyes  Closed, 
a)  3  second  segment  of  EEG  during  eyes  open  period,  b)  3  second  segment  of  EEG 
during  Eyes  Closed  period,  c)  EEG  spectrum  over  entire  60  seconds  of  Eyes  Open  period, 
d)  EEG  spectrum  over  entire  60  seconds  of  Eyes  Closed  Period.  Spectrum  plots  indicate 
moderate  alpha  wave  content  during  eyes  closed  and  alpha  suppression  during  eyes 

open. 
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Figure  62:  EEG  Characteristics,  at  C4-A2,for  Eyes  Open  Versus  Eyes  Closed, 
a)  3  second  segment  of  EEG  during  eyes  open  period,  b)  3  second  segment  of  EEG 
during  Eyes  Closed  period,  c)  EEG  spectrum  over  entire  60  seconds  of  Eyes  Open  period, 
d)  EEG  spectrum  over  entire  60  seconds  of  Eyes  Closed  Period.  Spectrum  plots  indicate 
minimal  differences  at  this  site  between  eyes  open  and  eyes  closed. 


Figure  63,  Figure  64,  and  Figure  65  show  measured  EEG  during  easy  driving  and  difficult 
driving  tasks,  at  the  same  three  electrode  locations  (T6,  P4,  and  C4).  These  plots 
demonstrate  a  dominance  of  low  frequency  activity  (<4  Hz)  during  both  easy  and  difficult 
driving,  with  greater  power  during  difficult  driving  for  all  electrode  locations. 

This  suggests  that  EEG  variations  occur  between  resting  and  driving  states,  as  well  as 
between  easy  and  difficult  driving.  The  increased  EEG  power  seen  during  difficult 
driving,  however,  may  be  due  to  the  increased  physical  activity  required  by  the  difficult 
road  which  had  more  curves.  Future  experiments  should  be  designed  to  examine  the 
difference  between  physical  loading  and  cognitive  loading  factors.  In  addition,  tasks 
which  involve  varied  cognitive  load  but  consistent  physical  load  can  be  examined  to 
remove  the  effect  of  varied  physical  loading.  These  methods  will  help  better  identify 
specific  indicators  of  cognitive  capacity  and  overload. 
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Figure  63:  EEG  Characteristics,  at  T6-A2,for  Easy  Driving  and  Difficult  Driving  Tasks, 
a)  3  second  segment  of EEG  during  Easy  Driving  period,  b)  3  second  segment  of  EEG 
during  Difficult  Driving  period,  c)  EEG  spectrum  over  2  V2  minutes  of  Easy  Driving,  d) 
EEG  spectrum  over  2  V2  minutes  of  Difficult  Driving.  Both  easy  and  difficult  driving  tasks 
are  dominated  by  Delta  wave  frequencies,  with  greater  power  during  difficult  driving. 
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Figure  64:  EEG  Characteristics,  at  P4-A2,  for  Easy  Driving  and  Difficult  Driving  Tasks, 
a)  3  second  segment  of EEG  during  Easy  Driving  period,  b)  3  second  segment  of  EEG 
during  Difficult  Driving  period,  c)  EEG  spectrum  over  2  V2  minutes  of  Easy  Driving,  d) 
EEG  spectrum  over  2  V2  minutes  of  Difficult  Driving.  Both  easy  and  difficult  driving  tasks 
are  dominated  by  Delta  wave  frequencies,  with  greater  power  seen  in  the  difficult 

driving. 
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Figure  65:  EEG  Characteristics,  at  C4-A2,  for  Easy  Driving  and  Difficult  Driving  Tasks, 
a)  3  second  segment  of  EEG  during  Easy  Driving  period,  b)  3  second  segment  of  EEG 
during  Difficult  Driving  period,  c)  EEG  spectrum  over  2  V2  minutes  of  Easy  Driving,  d) 
EEG  spectrum  over  2  V2  minutes  of  Difficult  Driving.  Both  easy  and  difficult  driving  tasks 
are  dominated  by  Delta  wave  frequencies,  with  greater  power  seen  in  the  difficult 

driving. 


Finally,  Figure  66  lists  average  heart  rate  and  average  respiration  rate  during  different 
stages  of  the  experiment.  These  results  indicate  no  significant  variation  in  either  heart 
rate  or  breathing  rate  between  the  different  stages  of  the  experiment.  It  is  possible  that 
longer  task  durations  may  be  required  to  observe  changes  in  these  parameters.  Also,  there 
can  be  variations  in  the  pulse  and  respiration  waveforms  that  are  not  reflected  in  the  rate. 


Figure  66:  Average  Heart  Rate  and  Average  Respiration  Rate  During  Experiment  Stages 
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For  example.  Figure  67  shows  a  representative  segment  of  the  respiration  waveform 
during  each  of  the  different  experiment  stages.  These  plots  show  that  during  ihc  resting 
stages  (eyes  open  or  eye  closed)  the  respiration  waveform  is  very  regular,  while  during 
the  driving  task  (easy  or  difficult)  the  waveform  is  irregular.  The  data  also  suggests  that 
breathing  may  be  more  erratic  during  the  more  difficult  task,  however  more  extensive 
testing  would  be  required  to  determine  sigmficance.  In  addition,  motion  may  be  partially 
contributing  to  the  irregularity  of  the  respiration  signal. 


0  5  10  15  20  25  30 
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b)  Respiration  -  Eyes  Closed 


Figure  67:  Respiration  Signal  During  Experiment  Stages. 

Regular  pattern  is  observed  during  resting  phases  (eyes  open  and  eyes  closed),  while  an 
irregular  pattern  is  observed  during  driving  tasks  (easy  and  difficult). 


7.2  HMMWV  Driving  Experiment 

As  another  preliminary  evaluation  and  assessment  of  the  HPBMS  concept,  we  conducted 
experiments  of  a  human  driving  task  using  a  HMMWV  on  a  standard  test  track.  This 
experiment  was  conducted  jointly  with  another  ARL  SBIR  project  which  w^  focused  on 
the  development  of  a  model  for  human  driving.  In  addition  to  providing  indications  of 
potential  benefit,  the  results  of  this  experiment  provided  initial  feedback  on  the 
operational,  procedural,  and  functional  issues  involved  in  the  execution  of  performance 
assessment  experiments  in  the  field,.  This  helped  guide  continued  development  and 
enhancement  of  the  HPBMS. 
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7.2.1  Experiment  Description 

The  HMMWV,  with  early  components  of  the  HPBMS  for  data  collection,  was  driven 
around  the  test  track  by  two  different  subjects,  for  several  trials  each.  'Die  subject  wore 
Cybemet’s  eye  tracking  goggles  to  provide  a  video  recording  of  the  driver’s  view, 
however  eye  tracking  was  not  performed  during  this  particular  experiment.  Electrodes 
were  placed  on  the  subject  to  provide  measurement  of  four  physiological  parameters: 

1)  Electrocardiogram  (EGG), 

2)  Electromyogram  (EMG)  of  the  upper  trapezius  muscle, 

3)  Electro-oculogram  (EOG)  for  horizontal  (left-right)  eye  motion,  and 

4)  EOG  for  vertical  (up-down)  eye  motion. 

Cybemet’s  portable  physiological  measurement  system  was  used  to  measure  and  capture 
the  electrode  signals. 

In  addition  to  the  video  of  the  driver’s  view  (recorded  on  VCR)  and  the  physiological 
signals,  data  from  the  vehicle  controls  (steering,  gas  pedal,  brake  pedal,  gear,  and  speed) 
were  also  recorded. 

Test  Track  Course 

The  test  track  course  was  prepared  by  ARL/HRED  personnel  to  be  similar  to  that  used  in 
previous  studies.  For  the  purposes  of  analysis,  the  track  is  divided  into  5  consecutive 
segments.  With  a  rest  stage  at  the  start  and  finish,  the  driving  task  consists  of  the 
following  stages,  in  the  order  presented  below: 

•  Initial  rest 

•  Straight 

•  Curves 

.  Tight  turns 

•  Cross  country 

•  Slalom 

•  End  rest 

Subjects  and  Trials 

Due  to  the  limited  time  available  to  conduct  the  sample  experiment,  two  subjects 
performed  the  driving  task  for  several  trials  each.  The  following  table  summarizes  the 
outcome  of  the  trials,  and  indicates  the  data  sets  that  were  used  for  physiological  signal 

analysis. 
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_ Outcome _ _ 

Subject  1 

Xrial  1  Successful  data  collection 

Xrial  2  Successful  data  collection 

Xrial  3  Successful  data  collection 

Trial  4  -  Trial  7  Electrode  signals  corrupted 


Data  Set  Label 


Subject#!  a 
Subject  #lb 
Subject  #lc 
Not  Analyzed 


Subject  2 

Xrial  1  Physiological  data  collection  error  Not  Analyzed 

Trial  2  Successful  data  collection  Subject  #2a 

Xrial  3  Successful  data  collection  Subject  #2b 

Xrial  4  Successful  data  collection _  Subject  #2c 

Figure  68:  Trial  Outcomes 


Data  Analysis,  ....  .  * 

The  data  was  analyzed  by  first  dividing  the  recorded  physiological  data  mto  the  different 

stages  of  the  driving  task.  This  was  performed  by  reviewing  the  video  tape  of  the  driver’s 
view  and  recording  the  time  of  transition  between  individual  stages  -  all  data  collected 
using  Cybernet’ s  Data  Collection  and  Analysis  Environment  (DCAE)  software  is  time 
synchronized. 

For  each  stage  of  driving  the  following  parameters  were  then  computed  based  on  the 
recorded  physiological  signals: 

•  Average  Heart  Rate 

•  Average  EMG  power 

.  Average  EOG  (left-right)  power  in  band  2Hz-50Hz 

•  Average  EOG  (up-down)  power  in  band  2Hz-50Hz 

Heart  rate  (HR)  was  derived  from  the  ECG  signal.  EMG  power  was  computed  from  the 
time  series  of  the  signal,  containing  frequency  components  from  30Hz  -  140Hz  (as 
filtered  by  the  physiological  measurement  system  hardware).  The  EOG  (left-right)  and 
EOG  (up-down)  signals  were  processed  by  first  deriving  the  frequency  spectrum,  and 
then  computing  the  power  contained  only  within  the  2Hz  to  50Hz  frequency  range. 

7.2.2  Experiment  Results 

The  tables  and  figures  that  follow  summarize  and  graphically  illustrate  the  results.  The 
data  suggest  significant  differences  in  all  physiological  parameters  measured  between 
different  stages  of  the  driving  task.  However,  the  experimental  procedure  was  not  robust 
enough  to  draw  statistically  significant  conclusions  -  a  larger  number  of  subjects  and  a 
more  controlled  experimental  procedure  would  be  required. 

Regardless,  this  work  achieved  significant  results.  The  effectiveness  of  the  HPBMS  data 
collection  systems  and  concept  for  workload  and  more  general  performance  assessment 
was  initially  demonstrated. 
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For  example,  the  results  reported  here  suggest  a  consistent  increase  in  heart  rate  as  the 
driving  task  becomes  more  difficult,  with  a  return  to  baseline  levels  after  the  driving  task 
is  completed.  Similarly,  EMG  power  (a  measure  of  muscle  tension)  shows  an  increase 
through  the  more  difficult  stages  of  the  driving  task,  and  a  return  to  baseline  levels  at  the 
end.  EOG  characteristics  also  suggest  a  trend,  with  the  horizontal  eye  motion  generally 
increasing  during  stages  with  more  turns,  and  vertical  eye  motion  decreasing  during  more 
difficult  stages.  We  hypothesize  that  this  decrease  in  the  vertical  EOG  is  primarily  due  to 
a  decrease  in  blink  rate,  since  the  EOG  (up-down)  signal  also  contains  components  due  to 
blink. 

Further  study  is  needed  to  better  identify  the  relationship  betweefi  these  (and  potentially 
other)  physiological  signals  and  performance/workload  parameters.  The  results  of  this 
work  will  be  used  to  help  define  future  research  and  experimental  efforts. 


Figure  69:  Summary  Data  from  Subject  #la 


Stage 

Duration 

Heart 

Rate 

Muscle 

Left-Right 

Up-Down 

Initial 

0:38 

75.8 

3.17E-08 

0:26 

77.1 

1.65E-09 

1.13E-08 

1.27E-08 

Curves 

0:34 

84.9 

2.32E-09 

2.41  E-08 

9.94E-09 

Turns 

0:42 

87.4 

2.16E-09 

3.31  E-08 

1.48E-08 

Cross  C. 

1:37 

90 

2.03E-09 

1.59E-081 

1.19E-08 

Slalom 

0:26 

90 

2.82E-09^ 

2.67E-08 

2.31  E-08 

End1 

0:23 

90 

1.16E-09 

3.95E-08 

End2 

0:23 

74.5 

8.95E-10 

Figure  70:  Summary  Data  from  Subject  #lb 


Stage 

Time 

Heart 

Rate 

Muscle 

Left-Right 

Up-Down 

initial 

0:30 

74 

1.28E-09 

1.36E-08 

3.36E-08 

0:22 

76.4 

1.31E-09 

9.69E-09 

8.25E-09 

Curves 

0:29 

84.8 

1.69E-09 

1.54E-08 

6.97E-09 

Turns 

0:39 

87 

2.1  IE-09 

2.81  E-08 

1.70E-08 

Cross  C. 

1:25 

81 

1.73E-0^ 

1.49E-0^ 

1.59E-08 

Slalom 

0:30  1 

88 

2.94E-09 

1.95E-08 

3.20E-08 

Endl 

0:15 

84 

1.33E-09 

1.75E-08 

5.31  E-08 

End2 

0:15 

72 

7.52E-10 
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Stage 

1’  / 

Time 

Heart  Rate 

Muscle 

Left-Right 

Up-Down 

Initial 

0:48 

72.5 

1.04E-09 

1.58E-08 

4.37E-08 

0:19 

82 

1.43E-09 

6.39E-09 

1.00E-08 

Curves 

0:29 

85.9 

2.12E-09 

2.09E-08 

1.19E-08 

Turns 

0:45 

84 

2.16E-09 

3.04E-08 

1.79E-08 

Cross  C. 

1:18 

2.00E-0^ 

1.64E-08 

1.12E-08 

Slalom 

0:30  ' 

90 

3.22E-09 

2.43E-08  1 

2.00E-08 

End1 

0:10 

90 

3.03E-09 

2.97E-08 

7.12E-08 

End2 

0:10 

78 

1.33E-09 

Fieure  72:  Summary  Data  from  Subject  #2a 


Stage 

Time 

Heart  Rate 

- ^ - 

Muscle 

uassMBM 

Initial 

0:27 

86.7 

2.16E-09 

1.35E-08 

8.57E-08 

Straight 

0:36 

80 

2.32E-09 

9.44E-09 

4.24E-08 

Curves 

0:52 

81 

3.02E-09 

1.80E-08 

2.98E-08 

Turns 

1:19 

85 

3.79E-09 

1.64E-08 

2.36E-08 

Cross  C. 

2:25  n 

81 

3.76E-09 

1.15E-0^ 

2.76E-08 

Slalom 

0:47 

93.5 

7.09E-09 

1.1  IE-08 

1 ,53E-08 

Endl 

0:10 

87.4 

9.75E-09 

2.54E-08 

7.94E-08 

End2 

0:11 

78.3 

3.66E-09 

Fi^re  73:  Summary  Data  from  Subject  #2b 


Stage 

J.  ^  f 

Time 

Heart  Rate 

Muscle 

Left-Right 

Up-Down 

Initial 

0:33 

79 

3.73E-09 

2.34E-08 

1.93E-07 

0:29 

80 

2.66E-09 

1.49E-08 

8.68E-08 

-  ^  - - 

Curves 

0:47 

82 

4.27E-09 

1.14E-08 

2.30E-08 

Turns 

1:15 

81 

4.41  E-09 

1.54E-081 

2.36E-08 

Cross  C. 

2:24 

84 

4.30E-09^ 

9.83E-09 

1.63E-08 

Slalom 

0:37 

91 

7.39E-09 

9.68E-09 

2.17E-08 

Endl 

0:10 

102 

7.64E-09 

1.78E-08 

6.98E-08 

End2 

0:07 

89 

1.53E-09 
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Figure  74:  Summary  Data  from  Subject  #2c 


Stage 

Time 

Heart  Rate  Muscle 

Left-Right  Up-Down 

Initial 

0:27 

87.8 

3.05E-09 

2.27E-08 

1.1  IE-07 

0:23 

84.8 

4.09E-09 

1.28E-08 

1.30E-07 

Curves 

0:28 

102.4 

4.96E-09 

1.16E-08 

4.38E-08 

Turns 

0:42 

101 

6.11E-09 

1.77E-08 

2.98E-08 

Cross  C. 

1:31  1 

100 

6.32E-09 

9.58E-09 

5.05E-08 

Slalom 

0:25 

113  n 

1.09E-^ 

1.35E-081 

4.38E-08 

End1 

0:12 

107 

6.85E-09 

2.30E-08 

2.79E-08 

End2 

7.3  Cognitive  Workioad/Computer  interface  Experiments 
7.3.1  Introduction 

This  set  of  experiments  was  conducted  toward  the  latter  part  of  the  Phase  11  effort,  once 
the  comprehensive  HPBMS  had  been  fully  integrated  and  validated.  These  experiments 
were  designed  to  exercise  every  feature  of  the  HPBMS  software  and  integrated  the 
hardware  components.  Additionally,  we  had  the  goal  of  testing  the  customization 
capabilities  of  the  system  -  for  example,  the  addition  of  new  hardware  (in  this  case,  extra 
TVs,  VCRs,  a  video  camera  and  a  third  computer  as  a  subject  station). 

As  a  result  of  sponsor  interest,  the  validation  experiments  were  designed  to  take  a  first 
step  towards  a  protocol  that  would  examine  the  effect  of  cognitive  worldoad  on  human 
performance  in  human  computer  interface  tasks.  The  specific  goal  of  this  set  of 
experiments  was  to  measure  and  analyze  human  performance-related  data  during  standard 
computer  interface  tasks,  in  order  to  achieve  both  a  demonstration  of  Ae  HPBMS 
capabilities,  and  to  conduct  a  preliminary  investigation  into  potential  indicators  of  human 

performance. 

The  result  of  this  experimental  protocol  was  highly  successful.  It  demonstrated  the  power 
of  the  developed  HPBMS  as  a  research  tool,  and  identified  specific  areas  for  continued 

research. 

Please  note  that  the  descriptions,  pictures,  and  instructions  given  here  may  vary  from 
the  final  version  of  the  HPBMS  hardware  and  software.  This  portion  of  the  report  is 
intended  to  provide  information  on  the  experimental  procedures  conducted,  not  to 
provide  instructional  directions.  Please  consult  the  HPBMS  documentation  for 
instructions  on  use  of  the  HPBMS. 


7.3.2  Background 

Detecting  and  recognizing  indicators  of  physical  workload,  and  based  on  those  indicators 
predicting  future  physical  performance,  is  perhaps  an  easy  task.  Most  people  will  make 
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predictions  of  this  sort  often  during  their  lives  -  either  predicting  their  own  performance 
or  predicting  the  performance  of  others.  And,  even  without  having  been  trained  to  know 
what  signs  to  look  for,  a  person  can  very  accurately  predict  their  own  physical 
performance  on  a  given  task,  and  to  a  lesser  degree,  the  performance  of  oftiers.  Those 
people  with  the  appropriate  training,  doctors,  athletic  coaches,  atiiletes,  will  have  a 
greater  degree  of  success  predicting  the  physical  performance  of  others. 

Clues  to  physical  performance  (obtained  without  external  hardware)  include;  breaths  per 
minute,  general  fatigue,  hunger  and  thirst.  Because  these  clues  are  easily  observable  in 
ourselves,  and  we  know  through  experience  how  we  perform  physically  when  breatWng 
at  a  certain  rate,  or  when  hungry,  we  can  gauge  the  performance  of  others  by  observing 
those  same  clues  -  the  rise  and  fall  of  a  chest,  amount  of  sweat,  speed  and  accuracy  of 
motion,  etc. 

However,  when  attempting  to  gauge  cognitive  performance  (decision  time,  accuracy  of 
answers,  good  vs.  bad  decisions),  the  signs  which  would  indicate  performance  are  absent 
from  casual  observations.  This  is  likely  do  to  the  fact  that  cognitive  processes  themselves 
cannot  be  casually  observed. 

Over  the  past  several  years,  there  have  been  many  new  techniques  developed  which  can 
be  used  to  search  for  the  hidden  indicators  of  cognitive  performance.  EEG,  the  reading  of 
electrical  signals  produced  by  the  brain,  watching  closely  the  movement  of  the  pupd,  and 
measuring  pupil  changes  in  size. .  .These  techniques  can  be  used  to  monitor  the  brain  md 
physiological  functions  (heart  rate,  etc....)  in  manners  not  possible  by  casual  observation. 

7.3.3  Hypothesis 

Because  the  human  brain  is  compartmentalized  in  terms  of  which  areas  are  vital  for 
which  tasks,  we  will  not  expect  to  find  a  single  physiological  indicator  for  workload  for 
all  types  of  tasks.  Visual  tasks  are  dealt  with  in  the  visual  cortex,  language  in  X,  memory 
in  Y,  etc. . . .  However,  this  does  not  mean  that  a  single  experiment  cannot  be  used  to  look 
for  workload  indicators  across  many  cognitive  venues  at  once. 


Motor  Cortex, 


Frontal  Lobe 


Lateral  Rssure-^l 
(Sylvius) 


Central  Fissure 

Sensory  Cortex 

Parietal  Lobe 

Occipital 
Lobe 


Cerebellum 


Temporal  Lobe-^ 

Figure  75:  Diagram  of  brain  sections 
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It  is  well  known  how  the  human  body  responds  to  physical  workload  -  sweat,  need  to  eat, 
drink,  heart  rate,  breathing  increase,  etc.  The  physicality  of  the  human  body  is  often 
compared  to  a  machine,  and  the  brain  to  a  computer.  There  exist  established  methods  for 
evaluating  and  exploring  the  operations  of  both.  If  the  human  body  displays  measurable 
reactions  to  physical  workload,  then  it  must  react  to  cognitive  workload.  Blood  flow  to 
different  brain  areas,  changes  in  electrical  activity  (EEG),  heart  rate  variability  changes, 
etc. 

Physical  signs  can  be  used  to  predict  future  physical  performance:  The  human  machine  is 
not  a  source  a  limitless  energy  -  in  a  given  period  of  time,  the  human  body  has  a  fimte 
source  of  nutrients  required  to  keep  it  going.  A  hard  physical  task  causes  an  expenditure  - 
of  energy,  depleting  the  energy  supply.  That  expenditure  is  observable  by  an  increased 
number  of  breaths  per  minute,  increased  heart  rate,  increased  sweat  production,  etc.  The 
increase  of  those  variables  are  a  measure  of  the  energy  expended  on  previous  tasks.  The 
more  energy  expended  on  previous  tasks,  as  indicated  by  the  amount  of  the  increase  in  the 
previous  factors,  leaves  less  available  energy  for  future  tasks.  Thus,  based  on  heart  rate, 
respiration  rate,  etc.,  we  can  predict  future  performance  based  (roughly)  on  the  amount  of 
energy  expended  during  previous  tasks. 

The  brain  can  be  thought  of  in  a  similar  manner:  the  brain  has  only  so  many  resources  to 
devote  to  processing  information.  When  information  is  being  processed,  those  resources 
are  being  used,  and  there  must  be  some  indicator  of  usage. 

7.3.4  Experiment  1  -  Recall,  Sternberg,  Tracking 

7.3.4.1  Background 

An  effective  methodology  for  workload  research  is  the  development  and  use  of 
standardized  tasks  for  calibrating  and  validating  mental  workload  measures.  The 
objective  is  to  provide  a  controlled  laboratory  procedure  which: 

•  Can  present  variable  degrees  of  workload  demand 

.  Is  well  understood  in  terms  of  subject  behavior,  repeatability,  and  statistical  variables 

•  Offers  easy  training  of  subjects  and  acquisition  of  data 

•  Permits  calibration  of  operator  workload  measures  against  norms 
.  Correlates  subjective  workload  with  workload  task  variables 

These  characteristics  make  such  simulated  tasks  highly  conducive  to  drawing  conclusions 
about  workload  effects  and  developing  psychophysiological  indicators  of  workload. 
Laboratory  methods  provide  the  highest  precision  of  measurement  in  the  realm  of 
operator  behavior.  The  researcher  has  control  over  the  performance  demands  imposed  on 
the  subject  and  the  test  procedures  can  be  easily  duplicated.  Tasks  can  be  varied  easily 
and  precisely  to  establish  the  sensitivity  of  task  measures  to  variables  of  known 
experimental  importance.  And,  safety  can  maximized. 
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Many  psychophysiological  workload  tests  have  been  developed,  implemented,  and 
reported  in  the  literature.  The  tests  selected  for  use  in  our  experiments  were  chosen  on  the 
basis  of  the  following  criteria;  1)  ease  of  use  by  the  researcher  and  subject,  2)  relevance 
of  the  exercised  cognitive  process  to  the  simulated  interface  application,  and  3)  likelihood 
of  the  procedure  to  offer  insight  into  the  assessment  of  workload  and  other  operator 
states.  Those  identified  include:  the  Sternberg  paradigm  (Sternberg,  1966  &  1969), 
number  recall,  and  object  tracking  tasks.  Each  of  these  task  can  be  uniquely  implemented 
to  more  closely  simulate  the  types  of  processes  required  in  the  tactical  command  and 
control  interface,  as  discussed  in  each  task  description. 


In  addition  to  examining  these  “basic”  tasks  independently,  we  also  examined  a  multi¬ 
task  scenario.  It  is  important  to  recognize  that  human  operators  typically  have  to  divide 
attention  between  more  than  one  task.  In  this  case,  the  operator  niust  often  postpone 
and/or  compromise  one  task  in  order  to  maintain  another.  It  is  this  division  of  attention 
between  tasks  in  a  complex  multi-task  interface  environment  that  we  would  like  to 
particularly  focus  on. 

Number  Recall  Task 

The  number  recall  task  tests  the  ability  of  a  subject  to  recall  material  from  his/her  short¬ 
term  sensory  store  (STSS)  and  short-term  memory  (STM).  In  its  simplest  form,  the 
subject  is  shown  a  number  consisting  of  (n)  digits  for  a  duration  of  (m)  seconds.  After 
this  presentation,  the  subject  is  then  requested  to  recall  the  number.  Typically  a  computer 
screen  and  keyboard  is  used  for  presentation  and  number  entry.  The  number  of  digits  (n) 
and  the  duration  of  presentation  (m)  can  be  adjusted  to  create  different  levels  of  cognitive 

demand. 

The  appearance  of  a  new  number  to  be  memorized  is  intermittent,  and  is  geiierally 
repeated  many  times  in  order  to  achieve  a  representative  average.  The  resulting 
performance  measures  for  this  task  are  the  number  of  digits  correct  and  the  response  time. 
The  subject  may  also  be  asked  to  recall  one  or  several  of  the  numbers  again  (without 
another  presentation)  at  some  later  time.  This  will  test  whether  the  subjects  STSS  went 
into  his/her  short-term  memory  (STM). 

This  procedure  can  be  used  to  test  a  subject’s  cognitive  capacity  for  STSS/STM  and 
examine  the  effects  of  increased  demand/workload. 

Many  variations  on  this  basic  theme  can  be  (and  have  been)  performed.  For  example, 
more  complex  information  than  simple  digits  may  be  used.  The  information  may  also  be 
presented  through  sound,  rather  than  sight,  in  order  to  assess  the  capability  to  recall 
auditory  information  versus  visual  information. 

In  regards  to  the  military  interface  application,  the  number  recall  task  can  be  used  to 
simulate  the  cognitive  process  involved  in  receiving  and  commimicating  various  types  of 
military  information  including  codes,  tactical  planning  information,  status  reports,  etc. 


143 


CYBERNET  SYSTEMS  CORPORATION 


Both  visual  and  auditory  forms  may  be  used  for  presentation  and  recall  of  the 
information. 

Sternberg  Paradigm 

Cognitive  ability  tests  have  had  a  long  history  of  use  by  large  orgamzatioris  for  employee 
because  of  their  validity  and  reliability.  In  addition  to  their  low  cost,  cognitive  test  have 
the  ability  to  predict  worker  performance  (Hunter  &  Hunter,  1984).  The  Sternberg 
paradigm  (Sternberg,  1966  &  1969)  is  often  used  for  testing  intermittent  and  response 

tasks. 

In  its  basic  implementation,  the  subject  must  memorize  a  list  of  (n)  target  digits  in  a  range 
of  0  to  9.  Then,  a  single  digit  in  the  0  to  9  range  is  presented  andlhe  subject  must  respond 
immediately  (by  pressing  a  button  or  speaking)  whether  or  not  the  number  is  among  the 
target  list.  The  appearance  of  a  set  of  new  target  numbers  (which  can  vary  in  length)  will 
be  intermittent,  and  is  typically  repeated  many  times  in  order  to  obtain  a  representative 

average. 

The  time  the  user  takes  to  enter  the  nximber  is  recorded  as  a  measure  of  his/her  response 
time;  the  performance  measure  for  this  task  is  typically  the  response  time  only,  with 
wrong  responses  ignored.  Reaction  time  has  been  found  to  be  dependent  on  the  number 
of  target  digits  and  the  number  of  concurrent  tasks.  The  more  targets  the  ^eater  the 
mental  demand,  which  increases  reaction  time.  The  subject  response  time  is  also 
dependent  on  skill  level  and  the  subject’s  sensitivity  to  training. 

The  Sternberg  reaction  time  is  a  good  calibrator  for  task  workload  since  its  demands  can 
be  easily  controlled  (usually  through  the  target  list  length)  and  for  its  easy  integration  into 
continuous  monitoring  tasks.  The  Sternberg  paradigm  has  real  world  applications,  as  it 
has  proven  useful  in  studying  Parkinson’s  Disease  (Schelosky,  et  al.,  1991),  as  well  as 
truck-driving  and  pilot  workload  (Rohrbaugh,  1997). 

Variations  on  this  basic  approach  in  order  to  simulate  the  military  interface  can  include 
using  more  complex  alphanumeric  or  symbolic  targets  rather  than  simple  digits.  Auditory 
cues  may  also  be  added  to  the  task.  Relevant  applications  include  recognition  of 
significant  targets  on  tactical  planning  maps  (i.e.  enemy  vs.  friendly  targets),  or  detection 
of  hostile  elements  in  reconnaissance  video. 

Tracking  Task  . 

Another  important  “basic”  task  involves  pursuit  and/or  compensatory  tracking  of  an 

object. 

In  pursuit  tracking  the  object  moves  randomly  within  a  two-dimensional  region  (typically 
on  a  computer  monitor).  The  subject  must  pursue  the  moving  object,  using  the  mouse  or 
other  input  device,  in  an  attempt  to  continuously  follow  the  path  of  the  object. 
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In  compensatory  tracking  the  object  attempts  to  move  around  the  screen  while  the  subject 
must  adjust  its  position  (using  the  mouse  or  other  input  device)  in  order  to  keep  it  within 
a  specific  region.  Maintaining  the  artificial  horizon  of  an  aircraft  is  an  example  of 
compensatory  tracking. 

Performance  in  ftie  tracking  task  is  measured  by  recording  the  error  in  each  diniension. 

The  target  region,  target  size,  lag  of  cursor  and  the  allowable  offset  can  be  modified  to 
create  varying  levels  of  the  difficulty.  Tracking  tasks  have  been  found  to  be  useful  in 
studying  Parkinson’s  Disease  (P.  Soliveri,  et  al.,  1997),  and  flight  simulation  (Roscoe,  et 
al.,  1981). 

The  tracking  task  offers  many  opportunities  for  variation  of  implementation  in  order  to 
simulate  various  real-world  environments.  Military  examples  of  applications  that  utilize 
tracking  skills  include:  weapon  targeting,  vehicle  driving,  aircraft  piloting,  object  pursuit, 
etc. 

Combining  Tasks 

In  many  real-world  environments  the  operator  must  perform  and  monitor  multiple  tasks 
simultaneously.  Thus,  it  is  often  necessary  to  evaluate  the  impact  of  different  task 
configurations,  so  that  optimal  performance  can  be  achieved.  The  multi-task  paradigm 
presents  the  subject  with  a  combination  of  tasks  to  be  operated  on  simultaneously,  and 
then  evaluates  the  way  in  which  the  subject  divides  attention  across  the  different  tasks 
and  how  this  affects  individual  task  and  total  task  performance.  The  “workload  test 
battery”  developed  by  the  Air  Force  to  assess  and  predict  pilot  performance,  uses  a  multi¬ 
task  paradigm. 

Using  a  similar  multi-task  paradigm  will  allow  Cybernet  to  gain  insight  into: 

(1)  Attentional  demands  of  each  task 

(2)  Difficulty  of  each  task  with  and  without  other  demands 

(3)  Optimum  display  configurations 

(4)  Consequences  of  cognitive  involvement  in  each  task 

(5)  Effects  of  time  stress 

(6)  Effects  of  anxiety  about  consequences 

The  attentional  demand  of  each  task  can  be  monitored  by  eye-tracking  which  records 
where  the  subject  is  looking.  The  complexity  and  difficulty  of  the  individual  tasks  can  be 
analyzed  by  subjective  evaluations  and  correlated  to  psychophysiological  measurements. 

Display  characteristics  can  be  changed  and  evaluated  to  see  what  features  contribute  to 
degraded  and/or  improved  performance. 

The  questions  this  type  of  experiment  aims  to  address  are  the  following.  Does  the  stress 
level  of  the  operator  increase  with  time?  What  happens  to  brain  wave  activity  when  the 
task  becomes  more  difficult  or  easier?  What  are  the  frequencies  of  cognitive  involvement 
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in  each  of  the  different  tasks?  Does  the  subject  become  frequently  overloaded?  How 
many  errors  does  he/she  make  during  the  tasks?  If  the  subject  is  anxious  about  the  ^ks 
does  this  hinder  hisAier  performance?  Does  it  increase  the  number  of  errors  that  he/she 
commits?  What  do  his/her  subjective  ratings  tell  about  their  anxiety  and/or  performance. 
Are  there  significant  psychophysiological  indicators  of  stress? 

Experiments  in  multi-tasking  may  by  their  nature  become  overly  elaborate  and 
cumbersome.  This  is  especially  true  when  the  experimenter  yields  to  the  temptation  of 
collecting  psychophysiological  information  about  every  task  variable  in  each  task.  It 
would  therefore  be  difficult  to  study  all  of  the  effects  for  every  variable  within  the  scope 
of  this  project,  since  there  are  simply  too  many  variables  involved.  Therefore,  certain  task 
variables  in  the  simulation  experiments  were  chosen  as  the  mosfappropnate  means  for 
collecting  psychophysiological  information  about  performance  and  cognitive  workload. 
The  effect  of  these  key  task  variables  are  examined  to  determine  their  effect  on  human 
performance  and  cognitive  processes. 


7.3.4.2  Hardware  Setup 

This  experiment  required  an  enhanced  version  of  the  standard  HPBMS.  The  additional 
hardware  pieces  were:  video  camera,  extra  TV  and  V CR  pair  (for  recording  and 
synchronizing  the  video  camera  output),  and  a  computer  and  monitor  on  which  to  run  the 
interface  experiments.  This  computer  served  as  the  ‘subject’s  station’.  The  ‘subject  s 
station’  and  ‘experiment’s  station’  has  been  explained  previously. 

The  16  channel  physiological  monitor  was  used  to  record  16  channels  of  EEG,  and  the  8 
channel  monitor  was  reserved  for  life  signals  and  an  event  marker  switch.  For  more 
information  on  the  exact  EEG  and  other  signals  monitored,  please  see  the  experimental 

procedure  below. 

7.3.4.3  Experimental  Parameters 

One  of  the  purposes  of  this  research  is  to  illustrate  how  numerous  specific  task  variables 
effect  human  performance  and  how  psychophysiological  measures  may  indicate  cognitive 
workload  or  degree  of  difficulty.  The  majority  of  the  differences  between  the  task 

difficulties  and  workload  may  be  attributed  to  cognitive  effects  (intellectual  or 

information-processing)  and/or  limitations.  Cognitive  limitations  can  be  can  be  linked  to 
the  limited  ability  to  project  the  effects  of  a  present  decision  into  the  future 
(predictability)  and/or  loss  of  decision  time  (reducing  delay  and  presentation  tunes). 

If  we  can  understand  and  model  the  human  performance  of  well-trained  subjects  in  these 
simple  laboratory  tasks,  then  perhaps  this  knowledge  may  be  extended  to  more  complex 
tasks  The  ability  to  repeat  laboratory  experiments  is  a  powerful  tool,  for  it  allows  us  to 
study  inter-subject  differences  and  the  effects  of  different  information,  and  it  provides  us 
with  a  measure  of  variability  inherent  in  a  human’s  cognitive  and  decision  process. 
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The  three  “basic  task”  operations  for  selected  for  our  experiment  were  discussed  in  the 
previous  section.  Here,  we  outline  the  parameters  of  each  task  procedure  that  may  be 
varied  in  our  experiment,  and  the  questions  they  can  address. 

Number  Recall 

Presentation  time 

1 .  Is  there  a  critical  presentation  time  required  for  accurate  memorization  and  recall? 

2.  How  brief  must  the  target  digits  be  presented  before  a  performance  decrement  is 
easily  recognizable? 

Delay  time 

1 .  Does  decreasing  the  delay  until  the  next  presentation  lead  to  a  performance  decrement 
and/or  increase  in  cognitive  load? 

2.  If  so,  how  short  should  the  delay  be  before  a  performance  decrement  is  easily 
recognizable? 

Number  of  disits 

1 .  What  is  the  limit  and/or  the  critical  number  of  digits  before  performance  drops  off 
dramatically? 

2.  If  the  digits  are  fixed  (i.e.  always  the  same)  does  the  task  become  more  predictable?  If 
so,  what  is  the  effect  when  compared  to  a  range  of  digits  (unpredictable)? 

Size/Font  of  disits 

1 .  What  happens  when  the  size  of  the  digits  is  decreased?  Increased? 

2.  Is  the  effect  of  size  dependent  on  the  users  ability  to  concentrate? 

3 .  Is  the  effect  of  size  dependent  on  the  users  vision? 

DMt  patterns 

1 .  Do  certain  patterns  of  digits  lead  to  easier  recall?  (i.e.  12345, 223334) 

Sternberg  Paradigm 

Presentation  time 

1 .  Is  there  a  critical  presentation  time  for  accurate  target  detection? 

2.  How  briefly  must  the  target  digits  be  presented  before  a  performance  decrement  is 
easily  recognizable? 

Delay  time 

1 .  Does  decreasing  the  delay  of  the  next  presentation  lead  to  a  performance  decrement 
and/or  increase  in  cognitive  load? 

2.  How  short  must  the  delay  be  before  a  performance  decrement  is  easily  recognizable? 
Number  of  digits 

1 .  What  is  the  limit  and/or  the  critical  number  of  target  digits  (if  any)  before 
performance  drops  off  dramatically. 
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2.  If  the  number  of  digits  is  fixed,  does  the  task  become  more  predictable?  If  so,  what  is 
the  effect  when  compared  to  a  range  of  possible  target  digits? 

Size/font  of  digits 

1 .  What  happens  when  the  size  of  the  digits  is  decreased?  Increased? 

2.  Is  the  effect  of  size  dependent  on  the  user’s  ability  to  concentrate? 

3.  Is  the  effect  ofsize  dependent  on  the  user’s  vision? 

Pursuit  Tracking 

Speed  of  object 

1 .  Is  there  a  critical  speed  at  which  performance  dramatically  decreases? 

2.  If  so,  is  it  subject  dependent  or  independent? 

Size  of  object 

1 .  W^at  happens  when  the  size  of  the  tracked  object  is  decreased?  Increased? 

2.  Is  die  effect  of  size  dependent  on  the  users  ability  to  concentrate? 

3 .  Is  the  effect  of  size  dependent  on  the  users  vision? 

Motion  of  object 

1 .  Does  a  more  predictable  non-random  motion  increase  performance? 

2.  What  type  of  motion  leads  to  the  greatest  performance  increase? 

Size  of  tolerance  resion 

1.  To  what  degree  does  tolerance  (i.e.  how  closely  the  object  must  be  tracked)  affect 
performance? 

2.  Will  changing  the  tolerance  region  lead  to  a  difference  in  cognitive  load? 

Cursor  shave 

1 .  Does  cursor  shape  (i.e.  a  cross-hair  versus  an  arrow)  affect  performance? 

Cursor  speed 

1 .  Does  the  g^  on  the  mouse  affect  performance? 

Duration  of  task 

1 .  If  the  testing  is  long  is  there  a  performance  decrement  as  time  goes  on? 

2.  Does  performance  fluctuate  with  boredom,  fatigue  or  distractions? 

3.  What  are  the  effects  of  short  vs.  long  tests? 

Compensatory  Tracking 

Speed  of  object 

1 .  Is  there  a  critical  speed  at  which  performance  drastically  decreases-  is  it  subject 
dependent  or  independent? 

Size  of  object 

1 .  What  happens  when  the  size  of  the  tracked  object  is  decreased?  Increased? 
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2.  Is  the  effect  of  size  dependent  on  the  users  ability  to  concentrate? 

3 .  Is  the  effect  of  size  dependent  on  the  users  vision?  - 

Size  of  tolerance  reeion 

1.  To  what  degree  does  tolerance  (i.e.  allowable  deviation  of  die  object  from  desired 
location)  effect  performance? 

2.  Will  changing  the  tolerance  region  lead  to  a  differences  in  cognitive  load? 

Duration  of  task 

1 .  If  the  testing  is  long  is  there  a  performance  decrement  as  time  goes  on? 

2.  What  are  the  effects  of  short  vs.  long  tests? 

Since  a  well  designed  experiment  should  only  have  a  few. variable  parameters  (in  order  to 
allow  for  effective  data  analysis),  we  selected  a  limited  set  of  the  above  parameters  to  be 
varied  in  our  experiments.  Selected  was  based  on  those  that  were  expected  to  have  the 
most  significant  effect  on  performance.  The  following  table  identifies  those  parameters. 


Task 

Task  variables/conditions 

Number  Recall 

Presentation  time 

Number  of  digits 

Sternberg 

Presentation  time 

Number  of  digits 

Pursuit  Tracking 

Speed  of  object 

Size  of  tolerance  region 

Compensatory  Tracking 

Speed  of  object 

Size  of  tolerance  region 

Figure  76:  Selected  Task  Variables  for  the  Number  Recall,  Sternberg  and  Tracking 

Tasks 

The  following  sections  specify  the  conditions  that  were  to  be  applied  for  the  set  of 
experiments  conducted. 

Number  Recall 

Values  for  Non-Variables: 
font  name=sansserif 
background  color=white 
font  style=plain 
loops=20 
digit  color=black 
font  size=26 
sound=off 


149 


CYBERNET  SYSTEMS  CORPORATION 


(1  ^  Condition  A:  Variable  =  Number  of  digits:  constant 


Number  Recall 

conjiguratio 
n  I 

conjiguratio 
n  2 

configuratio 
n  3 

minimum  digits 

3 

6 

9 

maximum  digits 

J) 

6 

9 

time  shown 
(sec) 

3 

'S 

time  between 
(sec) 

0 

2 

2 

(2)  Condition  B:  Variable  =  Number  of  digits:  variable 


conjiguratio 
111 _ 

2 


conjiguratio 
n  2 


Number 
Recall _ 

minimum 

digits _ 

maximum 

digits _ 

time  shown 
time  betireen 


*  optional 

Condition  C:  Variable  =  Presentation  time 


Number 
Recall _ 

time  shown 
time  between 
minimum 


configuration *  *configuratio 


configuration  configuration  configuration 


_ 

maximum 

digits 


*  less  than  1  second 


Sternberg 

Values  for  Non-Variables: 
font  nanie=sansserif 
background  color=white 
font  style=plain 
loops=20 
digit  color=black 
font  size=26 
sound=off 
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(1)  Condition  A:  Variable  =  Number  of  digits:  constant 


Sternberg 

configuration 

1 

configuration 

2 

configuration  \ 
3 

minimum 

5 

1 

digits 

maximum 

5 

1 

digits 

time  shown 

3' 

time  behveen 

2 

2 . 

2 

(2)  Condition  B:  Variable  =  Number  of  digits:  vanable 


Sternberg 


nununnm 
digits _ 

maximum 

digits 

time  shown 
time  between 


*  optional 


configuration  configuration  configuration 
2 _ _2 _ 3 _ 

2  5  8 


*confignratio 

n4 


Condition  C:  Presentation  time 


Sternberg 

configuration 

1 

configuration 

2 

configuration  \ 
3  \ 

time  shown 

2 

<1* 

time  between 

2 

2 

2 

minimum 

digits 

6 

6 

6 

maximum 

digits 

8 

8 

8  . . 

*  less  than  1  second 


Pursuit  Tracking 

Values  for  Non-Variables: 

fillColor=black 

fillMode=empty 

size=medium 

lineColor=black 

cursorSpeed=slow 

type=pursuit 

cursorShape=default 

backgroundColor=white 

testTime=2 

sound=off 


151 


CYBERNET  SYSTEMS  CORPORATION 


rnnHition  A:  Variable  =  Tolerance  region  I  • 


Pursuit 

configuration 

configuration 

configuration 

Tracking 

1 

2 

3 

tolerance 

40 

30 

20 

movement 

random 

random 

random 

speed 

moderate 

moderate 

m  Condition  B:  Variable  =  Speed:  tolerance  40 


Pursuit 

Tracking 

speed 

movement 

tolerance 


configuration  configuration  configuratioi 

1  _ 2 _ 3 _ 

moderate  _ fast  _  veO'fast _ _ 

random  _ random  random 

“io  40  40 


Condition  C:  Variable  =  Speed:  tolerance  30 


Pursuit 

Tracking 

configuration 

I 

configuration 

2 

configuration 

3 

speed 

slow 

moderately 

fast 

very  fast 

movement 

random 

random 

random 

tolerance 

30 

30 

r4^  Condition  D:  Variable  =  Motion  of  object 


Pursuit 

Tracking 

movement 

speed 

tolerance 


configuration 

J _ 

figure  8 

fast _ _ 

30 


configuration 

2 _ 

bounce _ 

fast 

_ 


configuration 

_i _ 

random 

fast 

30 


Compensatory  Tracking 

Values  for  Non-Variables: 

fillColor=black 

fillMode=empty 

size=medium 

shape=circle 

lineColor=black 

type=compensatory 

backgroundColor=white 

testTime=2 

sound=off 
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n  ^  Condition  A:  Variable  =  Tolerance  region  I:  moderate  speed 


Compensatory 

configuration 

configuration 

configuration 

Tracking 

1 

2 

3 

tolerance 

40 

30 

20 

movement 

random 

random 

random 

speed 

moderate 

moderate 

moderate 

(2)  Condition  B:  Variable  =  Tolerance  region  H:  moderately  fast  speed 


Compensatory  configuration  configuration  configuration 


Tracking 

tolerance 

movement 

speed 


2 _ 

50 _ 

random 

moderately 

fast 


2 _ 

_40 _ 

random 

moderately 

fast 
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30 

random 

moderately 

fast 


(3^  Condition  C:  Variable  =  Speed 


Compensator 

V  Tracking 
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configuration 

2 
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speed 

slow 
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fast 
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random 

random 

random 
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30 

30 

For  each  tracking  task,  the  user  was  required  to  track  the  object  for  two  minutes.  In  the 
number  recall  and  Sternberg  tasks,  20  iterations  of  each  configuration  was  utilized  to 
produce  a  total  (individual  configuration)  task  length  of  approximately  two  minutes. 

7.3.4.4  Procedure 

The  following  subject  consent  form,  volimtarily  signed  by  the  participant  before  the 
experiment  is  conduected,  outlines  the  experimental  procedure  conducted.  An  additional 
consent  form  for  video  recording  of  the  experiment  is  also  included. 

7.3.4.5  Subject  Consent  Form 

Project  #303_02  (IRB  0004.1) 


CONSENT  TO  INVESTIGATIONAL  TREATMENT  OR  PROCEDURE 

I _ ,  hereby  authorize  or  direct  the  primary  investigator,  and 

associates  or  assistants  of  his  choosing,  to  perform  the  following  treatment  or 
procedure  upon  myself: 
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This  Study  is  intended  to  assess  human  performance  and  examine  your  cognitive 
processing  state  when  performing  various  tasks  on  a  computer  interface  While 
performing  these  tasks  you  will  be  monitored  by  electrodes  to  measure  physiological  data, 
such  as  Electrooculogram  (EOG),  Electromyogram  (EMG)  and  Electrocardiogr^ 

(EGG)  Video  and  audio  will  be  recorded  during  testing.  You  will  be  wearing  a  Head 
Mounted  Eye  Tracking  device  or  a  Head  Mounted  Display  with  integrated  Eye  Tracker 
for  particular  parts  of  the  test. 

Confidentiality: 

Confidentiality  of  the  data  as  well  as  your  identity  will  be  maintained.  All  identi^ng 
information  about  yourself  will  not  be  shared  with  anyone  outside  of  CSC,  verbally  or  in 
writing,  without  the  permission  from  you.  Since  the  right  is  not  absolute,  there  are  some 
limits  to  confidentiality  you  should  know  about.  These  include: 

1  When  there  is  reasonable  suspicion  that  you  are  likely  to  harm  yourself  or  others.  If 
this  occurs  protective  measures  will  be  taken.  By  law,  these  situations  require  us  to 
reveal  necessary  information  to  persons  or  agencies. 

2.  If  your  records  are  subpoenaed  by  a  legitimate  court  order . 

3 .  Senior  staff  at  CSC  reserve  the  right  to  consult  at  CSC  in  staff  meetings  for 
consultation,  supervision,  or  educational  purposes. 

The  plans  for  future  use  of  data  as  part  of  this  study  or  use  beyond  this  study  will  be 
approved  by  an  IRB  (Internal  Review  Board)  committee  prior  to  use.  The  use  of  existing 
data  that  were  originally  obtained  for  different  purposes  ^d  that  involve  identifiable 
subject  information,  will  require  examination  of  the  risk  involved.  There  will  be  a 
determination  of  whether  the  new  use  is  within  the  scope  of  the  original  consent  or 
whether  necessary  or  feasible  to  obtain  additional  consent.  Anonymity  of  the  subjects 
must  be  preserved. 

The  signed  consent  forms  will  be  stored  in  the  contract  folder  at  CSC.  These  folders  are 
contained  in  locked  file  cabinets  and  only  authorized  personnel  are  allowed  to  handle 
these  documents.  Your  name  will  only  be  available  to  researchers  for  purposes  of  review 
of  subject  criteria  for  the  experiment.  Your  name  will  not  be  published  in  any  articles, 
journals  or  papers  without  your  written  consent. 


Data  Utilization 

•  Only  those  involved  in  the  project  #303  will  have  access  to  the  gathered  data 
.  Each  session  will  be  videotaped  and  audio  taped.  The  tapes  will  be  labeled  only  wiA 
a  code  number,  which  will  be  kept  in  the  investigator’s  files.  The  tapes  will  be  used  to 
obtain  brief  excerpts  for  supporting  data  analysis  and  to  illustrate  Cybemet’s  findings 
in  scientific  publications  or  at  professional  meetings. 
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•  Data  gathered  may  be  used  for  purposes  of  publication.  Your  name  will  be  kept 
confidential. 

Data  Records: 

Your  data  records,  such  as  consent,  are  maintained  in  a  locked  cabinet  at  CSC.  They  are 
not  part  of  any  shared  CSC  file,  and  no  one  has  access  to  them  except  CSC  staff.  The  data 
will  be  coded  to  remove  identifying  information. 

Data  that  could  reveal  your  identity  will  be  stored  in  files  accessible  only  to  the  primary 
investigator  and  authorized  staff. 

Questionnaires,  inventories  and  other  data-gathering  instruments  and  procedures  will  be 
carefully  designed  to  limit  the  personal  information  to  be  requested  only  tiiat  which  is 
essential. 

Federal  regulations  require  that  all  records  relating  to  the  IRB  and  to  human  subjects 
activities  be  retained  for  at  least  three  years  after  completion  of  the  research.  Records, 
including  signed  consent  form  and  collected  data,  must  be  accessible  for  inspection  at  any 
time  and  for  copying  by  authorized  representatives  of  Cybernet  or  the  agencies 
sponsoring  the  research. 

The  experimental  (research)  portion  of  the  treatment  or  procedure  is: 

The  general  procedure  will  be  to  instrument  you  with  recording  electrodes,  and 
then  have  you  perform  some  type  of  mental,  memory  and  tracking  task  involving  varying 
levels  of  cognitive  effort.  Physiological  data  will  be  recorded  in  synchrony  along  with 
task  performance  data.  Upon  playback,  we  will  be  able  to  analyze  the  task  data  and  note 
significant  cognitive  events.  We  will  then  study  the  physiological  data  recorded  during 
these  events  to  determine  which  parameters  exhibit  changes  that  could  be  used  as 
indicators  of  cognitive  workload  and/or  cognitive  capacity. 

The  following  is  the  procedure  the  investigators  will  follow  for  the  experimental  set-up 
and  collection. 

1)  Subject  reports  to  the  testing  facility  approximately  1/2  hour  before  the  experiment 
starts. 

2)  The  subject  will  fill  out  a  demographic  data  sheet  to  collect  pre-screening  data. 

The  demographic  data  will  determine  those  subjects  that  best  fit  Ae  selection  criteria. 
The  criteria  that  will  be  assessed  will  include:  if  the  subject  is  willing  to  volunteer 
for  the  testing,  whether  they  wear  glasses  and/or  contacts,  are  void  of  neurological 
disorders,  are  between  the  age  20-45  years  old,  are  not  pregnant,  and  if  they  have  no 
allergy  to  alcohol  or  tape.  If  the  subject  fails  to  meet  certain  criteria,  the  Investigator 


155 


CYBERNET  SYSTEMS  CORPORATION 


will  either  recommend  that  they  do  not  continue  with  the  experiment  or  dismiss  the 
subject. 

Time  :  estimated  5  minutes 

3)  Subject  will  be  informed  about  the  procedure  will  read  consent  form,  and  if  he/she 
wishes  to  participate  vwll  sign  the  consent  form.  Copies  of  the  Volunteer  Agreement 
Affidavit  consent  form  will  be  provided  to  the  subject  for  his/her  records. 

Time:  estimated  30  minutes 

4)  The  subject  will  be  briefed  on  using  the  computer  interfaces  as  well  as  the  other 
computer  peripherals. 

Time:  estimated  5  minutes 

5)  The  subject  will  practice  using  the  interfaces  for  up  to  5  minutes  for  each  of  the  3 
different  task  interfaces  to  become  accustomed  to  the  controls  and  display.  The 
subject  will  practice  until  a  certain  performance  criterion  is  met  (performance 
criterion  will  be  determined  at  a  later  time). 

Time:  estimated  15  minutes 

Electrodes  will  then  be  placed  on  the  subject  to  provide  measurement  of  the  desired 
physiologiceil  signals. 

6)  ECG  measurements  will  be  taken  via  electrodes  applied  to  the  chest. 

For  application  of  electrodes  the  following  will  occur: 

•  The  subject  will  remove  any  upper  clothing  ( i.e.  coat,  shirt)  to  allow  placement  of 
the  electrodes,  after  which  Ae  subject  may  put  the  clothing  back  on  for  the  test.  A 
private  room  or  area  will  be  available  for  placing  the  electrodes. 

•  The  skin  will  be  cleaned  at  electrode  locations  with  an  alcohol  swab  and  allowed  to 
dry. 

•  Two  electrodes  will  be  placed  in  precordial  position.  Precordial  leads  are:  VI ,  V2, 
V3,  V4,  V5,  and  V6.  These  six  measure  the  cardiac  electrical  current  in  an  anterior- 
posterior  aspect  with  regard  to  the  heart.  For  measurement  of  EKG  we  will  be  using 
the  V3  and  V4  positions  at  the  chest. 

The  following  figure  shows  the  placement  of  these  leads: 
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spaw  ^  rt^  of  stmwm: 


•  One  ground  electrode  will  be  placed  on  bony  area  (clavicle,  for  example). 

•  Electrodes  will  be  secured  with  surgical  tape. 

Time:  estimated  5  minutes 

7)  The  respiration  rate  will  be  measured  utilizing  a  band  placed  around  the  subjects 
abdomen  or  chest. 

8)  Dermal  skin  measurements  will  be  performed  utilizing  surface  electrodes  on  the 
finger. 

9)  EMG  electrodes  will  be  applied  to  upper  trapezius  (between  the  shoulder  blade  and 
the  neck)  as  shown  in  the  following  figure:: 


Electrode 


Figure  77;  EMG  location  of  the  Upper  Trapezius 
(From  Boston  University  Neurology,  Joe  F.  Fabre,  M.D.,  ©1997) 

•  Optimal  location  for  electrodes  are  determined  by  resisting  subject  s  shoulder 
elevation  while  feeling  for  the  flexing  trapezius  muscle  in  the  back  of  the  neck  area, 
(which  will  be  demonstrated  by  the  experimenter.) 

•  Area  is  marked  with  pen 
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•  Skin  is  cleaned  with  alcohol  swab  and  allowed  to  dry 

•  Electrodes  are  placed  and  secured  with  surgical  tape. 

Time:  estimated  4  minutes 


10)  EOG  electrodes  will  be  applied  to  the  face  as  shown  in  Figure  78: 


►  EOG  sensors  (Vertical)) 


Figure  78:  Typical  Placement  of  EOG  Sensors  for  Lateral  &  Vertical  Eye  Movements 
(From  McGuigan,  F.J.  (1979)  Psychophysiological  measurement  of  covert  oral  behavior:  A  guide 
for  the  laboratory.  Hillsdale,  NJ:  Lawrence  Erlbaum  Associates). 

•  Skin  will  be  cleaned  on  each  side  of  the  eyes  and  above/below  the  dominant  eye, 
using  alcohol. 

•  Four  electrodes  will  be  placed  and  secured  with  tape. 

Time:  estimated  3  minutes 

1 1)  The  EEG  cap  will  be  placed  on  the  subjects  head  with  electrode  positions  based  on 
the  International  10-20  System  (Figure  79).  Simply,  the  Electrodes  are  placed  on  the 
scalp  at  points  that  are  10  and  20  percent  of  the  distance  from  the  lines  of  the  nose  to 
the  back  of  the  head  (nasion  to  the  inion  in  Figure  79),  and  from  the  left  to  the  right 
mastoid  points  (just  behind  the  ear.) 
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Figure  79:  Electrode  Locations  in  the  International  10-20  System 
(From  Hippocrates  Project  NYU  School  of  Medicine,  1992-present) 


The  procedure  for  placing  the  cap  on  the  subject  is  as  follows: 

a)  Before  placing  the  EEG  cap  on  the  subject  the  neck  strap  and  chin  strap  are 
loosened 

b)  the  cap  is  placed  on  the  subjects  head  so  that  the  ears  fit  comfortably  through 
both  ear  loops 

c)  The  front  tab  is  adjusted  so  that  it  is  positioned  at  the  nasion  and  taped  in 
position 

d)  Chin  and  neck  strap  are  adjusted  so  that  they  are  centered  on  the  chin 

e)  The  EEG  cap  is  checked  to  ensure  that  the  placement  is  symmetrical 

f)  The  hair  is  parted  and  the  skin  under  the  biosensor  sockets  is  prepped  using  a 
cotton  swab  with  a  water  alcohol  mix. 

g)  The  biosensors  are  then  put  into  place 

12)  The  eye  tracker  will  be  placed  on  the  subject’s  head  over  the  EEG  cap,  adjusted  for 
comfort,  proper  imaging  of  eye,  and  the  correct  cameras  field  of  view. 

Time:  estimated  5  minutes 

13)  Wires  will  be  secured  with  tape  and  connected  to  the  physiological  monitor. 

Time;  estimated  2minut^ 

14)  All  signals  will  be  verified. 

Time:  estimated  15  minutes 

15)  The  EMG  signal  will  be  calibrated 

•  Shoulder  elevation  of  subject  in  upright  posture  will  be  resisted  by  holding  hand  down 

with  arm  along  side  of  the  body. 
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•  After  complete  relaxation  of  shoulder  subject  will  be  asked  to  build  up  elevation 
force  within  1-2  seconds  and  maintain  maximum  exertion  for  two  seconds. 

•  Data  will  be  recorded  simultaneously,  starting  while  subject  is  fully  relaxed,  one 
second  before  exertion. 

Time:  estimated  2  minutes 

16)  Eye  position  will  be  calibrated. 

•  Subject  will  gaze  at  fixed  points  while  eye  image  (and  EOG  data)  are  collected. 

•  Accurate  eye  tracking  will  be  verified  by  asking  the  subject  to  look  at  the  specific 
locations. 

Time:  estimated  5  minutes 

17)  The  subject  will  be  seated  at  the  workstation  and  cables  will  be  attached  and  arranged 
for  minimal  obstruction.  All  signals  will  be  re-verified  prior  to  beginning  the  tests. 

Time:  Up  to  10  minutes 

18)  The  subject  will  then  perform  the  series  of  tasks  while  all  data  is  measured  from  start 
to  finish.  After  each  task,  the  subject  will  be  asked  to  fill  out  a  short  form  reporting  on 
their  mental  and  physical  states.  A  session  will  consist  of  several  different  task  runs  with 
rest  periods  between.  Each  rest  period  will  last  approximately  3  minutes. 

The  order  in  which  the  subjects  perform  the  different  task  types  (number  recall, 
Sternberg  and  tracking  tasks)  will  be  randomized.  Future  experiments  will  also 
involve  interfaces  that  combine  multiple  tasks  into  a  single  interface.  The  exact 
jjjterface  parameters  and  configurations  to  be  tested  is  variable,  and  will  be 
determined  in  the  future. 

Time:  up  to  120  minutes 

Post-Test 

19)  At  the  end  of  the  experimental  procedure  the  test  participant's  cognitive  workload 
and  stress  levels  will  be  collected  with  the  NASA  TLX  and  the  SRE  forms.  These 
forms  will  be  filled  out  by  the  subject  once  the  last  test  has  been  completed. 

Time:  estimated  10  minutes 

20)  After  completion  of  the  entire  procedure,  the  eye  tracker,  electrodes,  and  lead  wires 
will  be  removed,  and  the  subject  will  be  allowed  to  fully  recuperate  for  up  to  30 
minutes,  imder  supervision,  to  monitor  if  any  complications  arise  from  the  testing 
(i.e.  dizziness,  tiredness  etc.). 
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21)  The  subject  will  be  de-briefed  and  given  a  point  of  contact  for  follow-up  on 
individual  performance  or  results  of  the  study. 

Time;  estimated  5  minutes 

Total  Time;  up  to  3  hours 

The  subject’s  participation  will  involve  1  to  3  visits  (on  different  days)  of  approximately 
3  hours  each. 


Purpose  of  the  procedure  or  treatment. 

The  goal  of  this  project  is  to  develop  a  system  that  will  facilitate  real-time  measurement 
of  human  performance  in  multi-task  environments.  This  will  be  performed  through  Ae 
measurement  and  analysis  of  psychophysiological-based  signals,  subjective  date  and 
direct  observation.  The  physiological  date  will  provide  insight  into  mental  workload, 
stress,  and  the  visual  cues  used  during  a  multi-task  situation. 


Benefits,  risks,  discomforts  and  precautions; 

There  is  no  real  direct  benefit  to  you.  Benefits  of  this  study  will  mostly  be  realized  by  the 
Army  through  the  development  of  a  cognitive  workload  and  performance  measures. 

There  will  be  a  potential  benefit  of  the  study  for  possible  expansion  into  future  projects  at 
CSC.  All  date  gathered  will  also  serve  to  enhance  the  system  for  use  as  a  scientific 
research  tool.  This  project  will  also  support  future  commercialization  of  the  system  whic 
may  provide  significant  benefits  to  a  variety  of  human  factors  research. 


The  risk  associated  with  this  set  of  experiments  is  minimal.  This  project  has  minimum 
risk  since  the  potential  harm  is  not  greater,  considering  probability  and  magnitude,  thm 
those  tasks  ordinarily  encountered  in  your  daily  life  or  during  the  performance  of  routine 
psychological  tests. 


Please  be  assured  that  the  MMDS  is  safe.  The  MMDS  will  be  optically  linked  to  the 
recording  computer.  This  provides  a  de-coupling  from  the  electrical  components  of  die 
computer,  thereby  eliminating  the  possibility  of  you  receiving  an  electneal  shock  while 

the  MMDS  is  attached. 


Although  all  instrumentation  is  non-invasive,  you  may  feel  discomfort  from  using  the  eye 
tracker  as  well  as  the  E-net  EEG  cap.  If  this  occurs  you  will  be  asked  if  you  wish  to 
continue,  delay,  postpone  or  discontinue  the  study. 

There  is  some  possibility  you  will  develop  simulator-sickness  like  symptoms  which  range 
from  slight  eye  strain  and  dizziness.  The  simulator  sickness  like  symptoms  may  happen 
when  wearing  the  HMD  or  eyetracker. 
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Any  incidents  of  simulator  sickness  will  be  followed  up  by  a  1  hour  observation  period 
during  which  time  we  do  not  wish  you  to  continue.  This  is  to  preclude  any  potential 
flashback  effects  of  simulator  sickness  which  have  been  known  to  occur  in  rare  cases. 

Due  to  the  investigational  nature  of  this  study  there  may  be  other  risks  that  are  currently 
unknown.  If  any  imforeseen  risk(s)  arise  from  the  project  the  IRB  committee  will  be 
informed  immediately. 

Procedure  For  Injuries: 

For  medical  emergency  or  any  personal  injury  to  you: 

•  Report  any  injuries  to  the  Investigator  as  soon  as  possible. 

•  The  Investigator  will  report  any  injuries  to  the  IRB  Committee  as  soon  as  possible 

•  An  injury  report  must  be  made  as  soon  as  possible  to  Cybernet  System  Corporation’s 
Human  Resovnce  office. 

•  Cybernet  Systems  Corp.  vrill  follow  all  OSHA  procedures. 

Criteria  for  participation  in  this  study: 

I  am  being  invited  to  participate  because: 

•  I  am  between  the  ages  of  20  and  45 . 

•  I  am  not  pregnant. 

•  lam  volunteering  for  this  experiment 

•  I  do  not  have  any  neurological  disorders  that  I  am  aware  of 

•  I  do  not  have  any  allergy  to  alcohol,  electrode  gel  or  tape  that  I  am  aware  of 

Anticipated  duration  of  subject’s  participation  (including  number  of  visits): 

Your  participation  will  involve  1  to  3  visits  of  approximately  3  hours  each. 

General  Conditions 

1 .  Should  you  consent  to  participate  in  this  research,  your  identity  will  be  kept 
confidential.  A  copy  of  the  informed  consent  documentation  will  be  made  available  to 
you  as  well  as  the  sponsoring  agency. 

2.  All  forms  of  research-  whether  routine  or  experimental  involve  some  minor  risk  of 
injury.  In  spite  of  all  precautions,  you  might  develop  dizziness  and  or  lethargy  from 
participating  in  this  project.  If  such  complications  arise,  the  researchers  will  provide 
you  with  assistance  for  obtaining  proper  rest. 

3.  If  in  the  event  you  develop  medical  complication  as  a  result  of  the  project  medical 
treatment  will  be  available.  However,  you  will  not  be  provided  with  reimbursement 
for  medical  care  other  than  what  your  insurance  carrier  may  provide,  nor  will  you 


162 


CYBERNET  SYSTEMS  CORPORATION 


receive  other  compensation.  CSC  has  made  no  provisions  for  payment  of  costs 
associated  with  injury  resulting  from  participation  in  this  project.  The  researchers  will 
assist  you  in  obtaining  appropriate  follow-up  medical  treatment  but  this  study  does 
not  provide  compensation  for  additional  medical  or  other  costs. 

4.  You  will  be  told  of  any  new  findings  that  may  influence  your  willingness  to  continue 
to  participate  in  the  project.  Your  participation  in  this  project  may  be  terminated  by 
the  project  investigator  if  in  his/her  judgment  it  is  inadvisable  for  you  to  contmue. 

5.  If  you  would  like  to  discuss  your  rights  as  a  research  subject  and  or  your  participation 
in  this  study  with  a  CRC  representative  who  is  not  part  of  this  study,  please  call 
Cybernet  at  734-668-2567  and  ask  for  Paulette  Hodge,  Human  Resources  Manager. 

6.  Should  you  agree  to  participate  in  this  research  you  may  change  your  mind  at  any 
time.  Refusal  to  participate  will  not  harm  your  relationship  with  the  facility  and  staff, 

nor  will  it  prejudice  further  relations. 

7.  COSTS:  Subject  compensation  will  not  be  provided  to  the  subjects.  Time  lost  and 
traveling  expenses  may  be  provided  to  subjects  outside  CSC. 


I  hereby  acknowledge  that  the  Investigator _ _ _ has  provided 

information  about  the  procedure  described  above,  about  my  rights  as  a  subject,  and 
he/she  has  answered  aU  questions  to  my  satisfaction,  I  understand  that  I  may 
contact  him/her  at  Phone  No.  (734)  668-2567  should  I  have  additional  questions. 
He/She  has  explained  the  risks  described  above  and  I  understand  them;  he/she  has 
also  offered  to  further  explain  all  possible  risks  or  complications. 

I  have  read  the  description  of  the  project  study  and  general  conditions  or  it  was 

read  to  me  by: _ _ •  Anything  I  did  not  understand  was 

explained  to  me  by  :  _ _ a»y  questions  I  had  were  answered 

by:_ _ _ 

I  understand  that,  where  appropriate,  the  U.S.  Army  may  inspect  records 
pertaining  to  this  study.  I  understand  further  that  records  obtained  during  my 
participation  in  this  study  that  may  contain  my  name  or  other  personal  identifiers 
may  be  made  available  to  the  sponsor  of  this  study,  the  U.S.  Army.  Beyond  this,  I 
understand  that  provisions  have  been  made  to  protect  my  privacy  and  to  maintain 
the  confidentially  of  data  acquired  through  this  research  project. 

I  understand  that  I  am  free  to  withdraw  my  consent  and  participation  in  this  project 
at  any  time  after  notifying  the  project  director  without  prejudicing  future  care.  No 
guarantee  has  been  given  to  me  concerning  this  procedure. 

I  have  not  given  up  any  of  my  legal  rights  or  released  the  investigator,  the  study 
sponsor,  the  facility  or  its  agents  from  liability  for  negligence. 
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I  have  read  and  fully  understand  the  consent  form.  I  sign  it  freely  and  voluntarily. 
On  this  basis  I  consent  to  participate  in  this  experimental  study.  I  have  been 
informed  that  a  copy  of  the  consent  form  will  be  given  to  me. 

I  certify  that  I  am/am  not  (circle  one)  participating  in  another  research  project  at 
this  time,  and  have  discussed  the  implications  of  such  activity  with  the  project 
investigators  of  this  project.  In  consideration  of  this  understanding,  I  voluntarily 
agree  to  participate  in  this  research  at  CSC. 


Print  Name _ _ _ _ 

Date:_ _ Time  AM  Signed _ 

PM  (subject) 


Witness _ _ _ _ _  _ _ _ _ _ _ — 

(person  authorized  to  consent  for 

subject,  if  required)  * 

*For  subjects  who  may  not  be  capable  of  providing  informed  consent  the  signature  of  a  legal  representative  is  required. 

I  certify  that  I  have  personally  overseen  the  completion  of  all  blanks  in  this  form 
and  explained  them  to  the  subject  or  his/her  representative  before  requesting  the 
subject  or  his/her  representative  to  sign  it. 

Signed^ _ ..  _ _ _ -  - - - 

(Signature  of  Project  Director  or  his/her 
Authorized  Representative) 

Witness: _ _ Date:  - - — 

Location: _ _ _ 
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7.3.4.6  Subject  Consent  to  be  Videotaped  Form 

Project  #  303_1  (IRB  0004-1) 


Video  Consent 

^^ith  your  permission,  you  will  be  videotaped  during  each  session  of  this  research.  Your 
personal  information  will  not  be  recorded  on  the  videotape  and  confidentiality  will  be 
strictly  maintained.  These  tapes  will  be  labeled  only  with  a  code  number,  which  will  be 
kept  in  the  investigator’s  files.  The  videotapes  will  be  kept  in  a  secured  cabinet  by  the 
Principal  Investigator  (PI)  of  this  study.  However,  you  should  be  aware  that  the  showing 
of  these  videotapes  may  result  in  others  being  able  to  recognize  you.  These  videotapes 
will  be  shown  under  Pi’s  direction  to  the  other  investigators,  associates,  and  other 
professionals. 

The  tapes  will  be  used  to  support  data  analysis  and  to  obtain  brief  excerpts  and  to 
illustrate  Cybemet’s  findings  in  scientific  publications  or  at  professional  meetings. 


If  you  agree  to  participate  in  this  study,  your  signature  on  this  video  consent  form  grants 
the  investigator(s)  permission  to  retain  the  videotapes  for  this  purpose.  However,  you 
have  the  right  to  review  the  videotapes  and  to  request  that  all  or  any  portion  of  the  tapes 
be  erased. 


I  give  permission  to  be  videotaped  for  this  research  project,  for  the  purposes  of  CSC 
research,  and  for  presentations  at  scientific  meetings  under  the  conditions  described. 


Print  Name 


Signature 


Date 


7.3.4.7  Results 

This  experiment  had  a  pool  of  six  (6)  subjects.  Complete  data  was  not  collected  fi:om 
Subject  2  because  the  subject  requested  the  experiment  be  aborted  part  way  through  due 
to  discomfort  of  the  EEG  cap.  In  subsequent  experiments  measures  were  taken  to  make 
the  EEG  cap  more  comfortable.  With  data  firom  only  5  subjects  it  was  deemed  difficult  to 
draw  statistically  significant  conclusions.  However,  described  below  are  the  results  from 
one  subject  which  provide  indications  that  there  are  potential  psycho-physiologicd 
indicators  of  individual  subject  workload.  In  our  analysis  looked  for  potential  indicators 
of  cognitive  workload  based  on  frequency  band  analysis  of  the  EEG  and  other  measures. 
The  following  tables  and  figures  show  results  and  generalized  indications  from  one 
particular  subject.  Similar  trends  were  seen  in  other  subjects. 
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Electrode  Pair 

Figure  8 
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Figure  81:  Power  Ratio  Theta/(Alpha+Beta) 
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-  -O" 

Easy  Recall 
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Hard  Recall 
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Slow  Tracking 
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Fast  Tracking 
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Figure  82:  Effect  of  Increased  Workload  in  the  Tracking  Task 


Parameter 

rr  - - 

Effect  of  increased 
workload 

Comments 

Beta  /  (Alpha  +  Theta) 

Possible  increase 

Some  electrodes 

Beta  /  Theta 

Possible  increase 

Some  electrodes 

Aloha  /  Beta 

Decrease 

Left  /  Right  -  alpha,  beta, 
delta 

Possible  decrease 

OP  /  CT  -  delta 

Decrease 

Other  narameters 

No  significant  change 
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Figure  83:  Effect  of  Increased  Workload  in  the  Sternberg  Task 


Parameter 

Effect  of  increased 
workload 

Comments 

Beta  /  (Alpha  +  Theta) 

Increase 

Except  random  cases  (4, 5) 
and  case  of  greatest  # 
digits  (3).  May  be  due  to 
two  opposing  effects. 

Beta  /  Theta 

Increase 

Same  as  above  ■ 

Aloha  /  Beta 

Decrease 

Left  /  Right  - 

Beta/(Alpha+Theta) _ 

Possible  decrease 

Left /Right -Beta/Theta 

Large  increase  for  fast 
presentation  (case  8) 

Others  unchanged 

OP/CT- 

Beta/CAlpha+Theta) 

Decrease 

OP  /  CT  -  Beta/Theta 

Large  increase  for  fast 
presentation  (case  8) 

Others  unchanged 

Other  parameters 

No  significant  change 

- - 

- ZZZ — - - - - 

Figure  84:  Effect  of  Increased  Workload  in  the  Number  Recall  Task 


Parameter 

Effect  of  increased 
workload 

Comments 

Beta/ (Alpha  +  Theta) 

Increase 

Strong  effect  due  to 
increased  #  digits. 

Minimal  effect  due  to 
timing  demand. 

Beta  /  Theta 

Increase 

Aloha  /  Beta 

Decrease 

Left/ Right -%Beta 

Decrease 

Timing  demand  only 

OP  /  CT  —  %  Gamma 

Increase 

OP  /  CT  ~  %  Beta 

Decrease 

Timing  demand  only 

Other  parameters 

No  significant  change 

_ _ _ _ _ _ 

Workload  rating  for  each  task  was  assessed  using  subjective  evaluation  methods.  The 
table  below  contains  the  results  of  the  NASA  TLX  “Task  Load  Indicator”  as  determine 
from  a  subject’s  self-reported  stress  and  workload  levels.  After  an  mdividud  tpk,  the 
subject  is  asked  to  rate  the  task  and  his  performance  based  on  a  scale  of  0-100  uj  terms  of 
how  important  a  particular  factor  was  in  performing  successfully  on  that  task.  Weights  ot 
importance  are  computed,  and  a  ‘Total  Workload’  value  is  obtained.  The  hi^er  this 
value  the  greater  the  amount  of  cognitive  workload  the  task  placed  on  the  subject.  We 
have  dso  provided  a  column  for  subject  reported  ‘Stress’  for  companson. 
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Figure  85:  NASA  TLX  Results  for  Self-Evaluated  Workload  and  Stress  of  Individual 

Tasks  .  _ 
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For  the  following,  slightly  more  detailed  analysis,  it  is  important  to  note  that  the  task  # 
(Recall  1,  Recall  2,  etc....)  indicates  the  relative  difficulty:  as  the  task  number  increases, 
the  difficulty  of  the  task  increases  as  well. 

The  below  figure  demonstrates  the  total  EEG  Theta  band  power  for  all  the  listed  tasks.  It 
was  indicative  of  all  tasks  to  show  repressed  EEG  power  from  baseline  (non-moving, 
non-stimulated,  eyes  open).  In  the  below  example  we  can  see  that  as  difficulty  increases 
in  both  the  Recall  and  Tracking  tasks,  EEG  Theta  activity  is  further  suppressed.  Results 
for  the  Sternberg  tasks  may  at  first  appear  anomalous,  but  remember  that  Sternberg  l  ^ 
&  5,  and  6-8  are  separate  sub-groupings.  Looking  at  this  we  notice  a  decrease  in  activity 
across  tasks  1-3,  a  stable  level  of  activity  for  tasks  4  &  5  (our  random  cases),  and 
decrease  in  activity  for  tasks  6-8. 
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Figure  86:  EEG  Theta  Band  Power 


The  next  figure  below  displays  the  %  that  the  Theta  band  represents  in  terms  of  total  EEG 
power.  Here  we  can  see  similar  trends  to  what  was  shown  above.  As  workload  increases 
for  the  Recall  and  Tracking  tasks.  Theta  activity  is  suppressed.  However,  for  the 
Sternberg  tasks,  as  workload  increases  (tasks  1-5)  %  Theta  increases.  The  interesting 
cases  are  Sternberg  tasks  6-8,  where  %  Theta  activity  decreases.  These  tasks  increased 
workload  by  decreasing  the  amount  of  time  the  subject  had  in  which  to  function.  This 
suggests  that  mental  workload  due  to  cognitive  factors,  and  mental  workload  due  to  time 
constraints  may  have  separate  physiological  indicators.  Tracking  task  7  is  considered  to 
be  anomalous,  possibly  caused  by  a  poorly  seated  electrode  or  wire  connection. 


Figure  87:  EEG  %Theta  Power 
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The  figure  below  is  a  plot  of  heart  rate  Vs  workload  (for  .all  tasks).  You  can  see  that  there 
is  a  general  trend  for  heart  rate  to  increase  as  workload  increases. 


Figure  88:  Heart  Rate  Vs  Workload 


7.3.4.8  Discussion 

Because  of  the  small  subject  pool,  it  was  determined  that  there  was  not  enough  data 
collected  to  provide  for  any  statistically  valid  conclusions.  However,  as  the  above 
selected  results  show,  there  are  many  potential  indicators  of  cognitive  workload. 

The  next  step  in  a  study  of  this  sort  would  be  to  examine  the  data  for  the  strongest 
possible  indicators,  then  refine  the  experimental  procedure  for  speed  and  enhanced  data 
collection  around  those  parameters.  A  collection  of  roughly  10  parameters  would  be 
chosen,  along  lines  for  strength  of  signal  and  repeatability  across  the  subject  pool.  The 
experimental  tasks  provided  in  the  next  experiment  should  better  reflect  the  workload 
parameters  we  are  interested  in  examining,  as  well  as  producing  the  types  of  cognitive 
workload  that  we  have  determined  are  the  most  interesting  to  study. 


7.3.5  Experiment  2  -  Tetris,  Reading,  Mensa 

7.3.5.1  Background 

By  varying  cognitive  workload,  we  can  effect  changes  in  Task  Performance  (for  both 
cognitive  and  physical  ability)  and  Physiological  Measures.  By  varying  physical 
workload,  we  can  effect  changes  in  Task  Performance  (for  both  cognitive  and  physical 
ability)  and  Physiological  Measures.  Having  categorized  these  changes  and  resulting 
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patterns,  we  can  look  for  these  changes  and  patterns  as  signs  of  degrading  cognitive  and 
physical  performance.  In  addition,  based  on  the  parameters  designed  in  the  experiments, 
we  can  use  our  knowledge  of  those  parameters  to  analyze  specific  work  environments  to 
provide  a  ‘Degradation  Number’  to  the  environment.  This  ‘Degradation  Number’  would 
be  an  indicator  of  how  strongly  that  operational  environment  would  adversely  effect 
operator  performance. 

This  experiment  was  developed  both  to  provide  the  HPBMS  with  a  comprehensive  ‘real’ 
scenario  under  which  to  test  the  functionality  of  the  system  after  upgrades  and 
improvements  had  been  made  following  the  previous  set  of  validation  experiments.  In 
keeping  with  the  goals  of  tiie  previous  experiment,  and  building  on  the  results  of  data 
analysis,  new  computer  interface  tasks  (and  other  cognitive  tasks)  were  developed  for  use. 
These  new  tasks  would  be  more  engaging  both  in  terms  of  hand-eye  activity  as  well  as 
keeping  the  subject  continuously  cognitively  challenged.  They  were  designed  to  move 
away  from  the  more  memory  related  tasks  of  the  first  experiment. 

Problems  encoimtered  with  the  previous  set  of  tasks  were  that  they  were  boring.  Subjects 
indicated  that  their  boredom  and  lack  of  cognitive  enthusiasm  was  a  possible  contribution 

to  errors. 

7.3.5.2  Hardware  Setup 

This  experiment  required  an  enhanced  version  of  the  standard  HPBMS.  The  additional 
hardware  pieces  were;  video  camera,  extra  TV  and  VCR  pair  (for  recording  and 
synchronizing  the  video  camera  output),  and  a  computer  and  monitor  on  which  to  run  the 
interface  experiments.  This  computer  served  as  the  ‘subject’s  station’.  The  ‘subject  s 
station’  and  ‘experiment’s  station’  has  been  explained  previously. 

The  16  channel  physiological  monitor  was  used  to  record  16  channels  of  EEG,  and  the  8 
channel  monitor  was  reserved  for  life  signals  and  an  event  marker  switch.  For  more 
information  on  the  exact  EEG  and  other  signals  monitored,  please  see  the  experimental 

procedure  below. 

7.3.5.3  Experimental  Parameters 

Experimental  tasks  can  be  designed  along  3  major  variables:  time  constraints,  cognitive 
difficulty,  and  separation  of  attention  (based  on  the  number  of  tasks  needing  to  be 
processed  at  a  single  time  -  multitasking). 

While  temporal  constraints  can  greatly  effect  performance,  in  an  effort  to  limit  the 
number  of  variables  present  in  the  experiment,  those  co^traints  have  been  removed  m 
favor  of  being  able  to  more  strongly  characterize  cognitive  workload  effects  and 
separation  and  demands  of  attention.  Several  new  tasks  were  introduced  for  this 
experiment.  The  following  identifies  the  parameters  of  interest  for  these  tasks. 

Tasks: 

Tetris 
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Gaze  Location  =  where  the  subject  is  looking  and  how  long  they  look  at  certain 
locations  will  be  key  in  determining  where  their  focus  of  attention  is: 
preview  box  (cognitive  functions),  keyboard  (physical  functions), 
‘waterfall  area’  (cognitive  functions). 

Time  =  to  task  completion 

Variables  =  #  of  games  running  simultaneously 

#  of  lines  to  be  completed  in  each  Tetris  screen 
hnportance  values  (for  multiple  game  scenarios  only) 

Win  Conditions  =  complete  X  number  of  lines  in  a  single  game 

complete  X  number  of  lines  in  each  of  A  simultaneously  running  games 
Multiple  games,  where  each  has  an  importance  value  associated  with  it 
(completion  of  X 

lines  is  the  goal  here  as  well) 

Performance  =  number  of  lines  completed  vs.  Importance  value  vs.  Number  of 
lines  required 

Brain  Teasers  (Mensa  Test) 

Gaze  Location  =  given  the  nature  of  the  task,  gaze  location  could  be  irrelevant 
Time  =  to  complete  the  teaser 
Variables  =  difficulty  level  of  the  puzzles 

#  of  puzzles 

Win  Conditions  =  to  solve  the  puzzle 

Performance  ~  time  to  complete  the  puzzle  vs.  Correctness  vs.  Puzzle  difficulty 
Phone  Interruptions 

Gaze  Location  =  time  spent  looking  at  certain  locations  while  engaged  in 
conversation 

Time  =  time  is.  likely  to  be  irrelevant  in  this  task  or  time  to  write  down  the 
information  supplied 

Variables  =  number  of  calls  per  time  period 

amoimt  of  information  asked  or  supplied  for  record 
difficulty  of  information  supplied  or  questions  asked 
length  of  time  of  phone  calls 

Win  Conditions  =  to  accurately  record  all  information  given  over  the  phone 
to  accurately  answer  all  questions  asked  over  the  phone 
Performance  =  time  to  answer  questions/record  information  vs.  Correctness  of 
answers  or  Accuracy  of  recording  vs.  Variability  of  calls 


7.3.5.4  Procedure 

The  experimental  procedure  for  this  experiment  was  exactly  the  same  as  for  the  one 
previously  described  (see  Section  7.3. 4.4)  with  the  exception  that  a  different  set  of  tasks 
was  performed.  The  following  outline  describes  the  tasks  that  were  performed: 
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1  ■  Collect  data  while  the  subject  reads  a  passage  of  text  (“ReadBase”) 

The  subject  was  asked  to  sit  in  a  chair,  relax,  and  read  aloud  the  text  on  a  sheet  of  paper. 
The  passage  took  approximately  5  minutes  to  read 

2.  Collect  data  while  the  subject  performs  the  compensatory  tracking  task  (“TrackBase’D 

The  subject  performed  the  same  tracking  task  seen  in  the  previous  set  of  experiments, 
using  the  default  parameter  values 

3.  Collect  data  while  the  subject  plays  a  game  of  Tetris  (“TetrisBase”) 

The  subject  played  a  single  game  of  tetris  (standard  computer  game  involving  the 
placement  of  falling  bricks  of  different  shapes  and  orientation).  The  task  lasted  as  long  as 
the  subject  could  keep  the  game  from  ending. 

4.  Collect  data  while  the  subject  solves  a  set  of  brain  teaser  questions  (“MensaBase”) 

The  subject  was  asked  to  answer  as  many  of  the  questions  correctly  as  possible  within  the 
alotted  10  minutes  of  time.  They  were  not  given  any  feedback  as  to  how  much  time  was 
remaining.  The  questions  were  taken  from  a  Mensa  Test  Puzzle  book. 

5.  Collect  data  while  the  subject  plavs  two  simultaneous  games  of  Tetris  (“2Tetris!;) 

The  subject  played  2  games  of  Tetris  at  once  with  the  goal  of  maximizing  total  score.  The 
two  screens  of  Tetris  were  setup  on  the  screen  side  by  side,  with  a  1cm  gap  between 
them.  The  subject  had  to  use  the  mouse  to  switch  control  between  the  two  screens.  The 
task  continued  until  both  games  were  completed. 

6.  Collect  data  while  the  subject  manages  multiple  different  tasks  simultaneously 
r^Multask’T 

This  collection  involved  the  simultaneous  operation  of  a  tracking  task,  a  game  of  tetris, 
and  the  addition  of  interrupting  phone  calls  that  demanded  cognitive  attention.  The  phone 
calls  arrived  very  30  seconds  (roughly)  and  the  subject  was  required  to  answer  the  phone 
as  quickly  as  possible  and  answer  the  question  posed  by  the  caller.  Questions  involved 
simple  mathematical  operations  and  recall  of  previously  stated  nonsense  information. 

This  tasks  lasted  a  total  of  6  minutes.  If  for  some  reason  the  Tetris  game  ended  during 
this  time,  a  new  game  was  immediately  begun  and  noted  in  the  experiment  log. 

7.  Collect  post  experiment  resting  data  (“PostMult”) 

At  the  end  of  the  experiment,  5  minutes  of  data  was  collected  while  the  subject  rested  in  a 
chair  with  eyes  open. 
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7.3.6  Analysis 

The  experiment  had  a  pool  of  three  (3)  subjects.  Data  was  not  collected  from  one  of  the 
subjects  because  the  experiment  was  aborted  due  to  system  difficulties.  Described  below 
are  the  results  from  one  subject  which  provide  indications  that  there  are  physiological 
indicators  of  individual  subject  workload. 

Below  are  a  selection  of  some  of  the  data  plots  used  in  the  analysis  which  most  strongly 
suggest  physiological  indicators  that  shoxild  be  examined  further  in  future  experiments. 
The  data  for  the  subject  playing  two  games  of  Tetris  at  once  was  lost  due  to  a  battery 
failure  with  the  physiological  monitors. 

Below  is  a  graph  of  the  EEG  channels  in  the  Alpha  frequency  band.  All  levels  are 
depressed  from  the  reading  task  ('ReadBase  )  except  for  the  brain  teaser  task 
(‘Mensabase’).  It  is  apparent  that  EEG  in  the  Alpha  band  is  related  to  strong  cognitive 
workload  -  during  the  Mensa  test,  the  most  cogmtive  challenging  of  all  the  experimental 
tasks,  EEG  in  the  Alpha  band  was  the  highest.  Following  the  trend  from  most  cognitively 
challenging  to  the  least,  we  find  that  ‘Multask’,  with  it’s  phone  calls  and  questions  was 
the  next  most  challenging,  then  down  to  playing  Tetris,  and  lastly  to  the  Tracking  task, 
which  required  no  or  little  cognitive  thought  at  all,  being  mostly  a  hand-eye  coordination 

task. 


Figure  89:  EEG  Power  Levels  in  the  Alpha  Band 


This  data  demonstrates  that  in  times  of  strong  cognitive  workload,  the  EEG  i^pha  band 
will  likely  be  a  strong  indicator  of  that  workload,  and  possibly  even  of  cognitive  burnout. 
(This  would  need  to  be  evaluated  in  future  experiments.) 
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The  data  plot  below  shows  the  subject’s  average  heart  rate  across  the  different 
experimental  trials.  With  an  increase  in  task  cognitive  difficulty,  the  subject  experiences 
an  increased  heart  rate. 
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Figure  90:  Subject  Heart  Rate 


The  table  below  contains  the  results  of  the  NASA  TLX  “Task  Load  Indicator”  as 
determined  from  a  subject’s  self-reported  stress  and  workload  levels.  After  an  individual 
task,  the  subject  is  asked  to  rate  the  task  and  his  performance  based  on  a  scale  of  0-100  in 
terms  of  how  important  a  particular  factor  was  in  performing  successfully  on  that  task. 
Weights  of  importance  are  computed,  and  a  ‘Total  Workload’  value  is  obtained.  The 
higher  this  value,  the  greater  the  amount  of  cognitive  workload  the  task  placed  on  the 
subject.  We  have  also  provided  a  column  for  subject  reported  ‘Stress’  for  comparison. 
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Figure  91:  NASA  TLX  Index 


7.3.7  Discussion 

Because  of  the  small  subject  pool,  it  was  determined  that  there  was  not  enough  data 
collected  to  provide  for  any  statistically  valid  conclusions.  However,  as  the  above 
selected  results  show,  there  exist  potential  indicators  of  cognitive  workload. 

The  next  step  in  a  study  of  this  sort  would  be  to  examine  the  data  for  the  strongest 
possible  indicators,  then  refine  the  experimental  procedure  for  speed  and  enhanced  data 
collection  around  those  parameters.  A  collection  of  roughly  10  parameters  would  be 
chosen  along  lines  for  strength  of  signal  and  repeatability  across  the  subject  pool  The 
experimental  tasks  provided  in  the  next  experiment  should  better  reflect  the  workload 
parameters  we  are  interested  in  examining,  as  well  as  producing  the  types  of  cognitive 
workload  that  we  have  determined  are  the  most  interesting  to  study. 


7.3.8  Appendix  of  Experimental  Questionnaires 

7.3.8.1  Demographic  Data  Sheet 

Project  #  303_2  (IRB  0004-1) 

ID;_ _ 

Demographic  Data  Sheet 
PERSONAL  QUESTIONNAIRE 

Please  provide  us  with  the  following  information,  so  that  we  can  better  evaluate  our  data 
and  protocols.  Circle  the  choice  that  applies: 
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Are  you  a  willing  volimteer  for  testing  ? 

Yes 

No 

Have  you  been  a  volunteer  for  an  experiment  before  ? 

Yes 

No 

Do  you  wear  glasses  and/or  contacts  ? 

If  so,  which  one(s) _ _ _ 

Yes 

No 

Do  you  wear  corrective  lenses  for  reading  ? 

Yes 

No 

Are  you  void  of  all  neurological  disorders  ? 

(i.e.  epilepsy,  Alzheimer’s,  etc.) 

Yes 

No 

Are  you  pregnant  (females  only)  ? 

Yes 

No 

Do  you  have  any  allergy  to  alcohol  or  tape  ? 

Yes 

No 

Eye  Dominance  (if  not  known  leave  blank) 

left 

right 

Have  you  recently  hurt  or  injured  any  of  the  following? 

(Circle  all  that  apply): 

Left  Hand  Right  Hand  Legs 

Head 

Neck 

Left  Arm  Right  Arm  Back 

Eyes 

Chest/Torso 

Age 

Years  Computer  Experience: 

0  1-3  3-5 

Highest  Degree  Attained 


5+ 


H.S.  Diploma 
Bachelors  Degree 
Masters  Degree 
Ph.D. 

Post-doctorate 

M.D. 

None 
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7.3.8.2  Satisfaction  Survey 

Project  #  303_1  (IRB  0004-1) 

Satisfaction  Survey 

Please  help  us  evaluate  our  procedures  by  answering  some  questions  about  your 
experience.  We  are  interested  in  your  honest  answers  both  positive  or  negative.  Your 
name  will  be  kept  confidential  we  are  looking  at  overall  feedback  and  not  individual 
responses.  Please  do  not  put  your  name  on  this  form. 

Circle  the  response  that  best  fits  your  answer  to  the  following  questions: 

1.  How  satisfied  were  you  with  the  topics  covered  by  the  consent  form  and  investigators? 

•  Very  Satisfied 

•  Somewhat  Satisfied 

•  Neither 

•  Slightly  Dissatisfied 

•  Very  Dissatisfied 

2.  How  satisfied  were  you  with  the  amount  of  information  supplied  to  you? 

•  Very  Satisfied 

•  Somewhat  Satisfied 

•  Neither 

•  Slightly  Dissatisfied 

•  Very  Dissatisfied 

3.  How  difficult  was  it  for  you  to  remain  comfortable  during  testing? 

•  Very  Difficult 

•  Somewhat  Difficult 

•  OK 

•  Pretty  Easy 

•  Not  a  Problem 

4.  How  satisfied  were  you  with  the  availability  of  staff  when  you  needed  help? 

•  Very  Satisfied 

•  Somewhat  Satisfied 

•  Neither 

•  Slightly  Dissatisfied 

•  Very  Dissatisfied 
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5.  Please  rate  your  satisfaction  with  the  investigator(s)  in  each  of  the  following  areas. 

5a.  Knowledge  of  the  subject 

•  Very  Satisfied 

•  Somewhat  Satisfied 

•  Neither 

•  Slightly  Dissatisfied 

•  Very  Dissatisfied 

5b.  Preparation 

•  Very  Satisfied 

•  Somewhat  Satisfied 

•  Neither 

•  Slightly  Dissatisfied 

•  Very  Dissatisfied 

6.  Were  you  treated  with  courtesy  and  respect  by  CSC? 

•  Very  Satisfied 

•  Somewhat  Satisfied 

•  Neither 

•  Slightly  Dissatisfied 

•  Very  Dissatisfied 

7.  How  satisfied  were  you  with  the  sensitivity  of  maintaining  confidentiality? 

•  Very  Satisfied 

•  Somewhat  Satisfied 

•  Neither 

•  Slightly  Dissatisfied 

•  Very  Dissatisfied 

8.  Generally,  how  satisfied  were  you  with  the  helpfulness  of  this  procedure  in  answering 
all  your  questions? 

•  Very  Satisfied 

•  Somewhat  Satisfied 

•  Neither 

•  Slightly  Dissatisfied 

•  Very  Dissatisfied 
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9.  How  many  of  the  things  you've  learned  through  this  study  do  you  think  you  will  use  in 
everyday  life? 

•  Most 

•  Some 

•  Only  a  Few 

•  None 

•  Don't  Know 

10.  How  much  did  you  enjoy  doing  the  experiment? 

•  Really  Enjoyed 

•  Somewhat  Enjoyed 

•  Neutral 

•  Did  Not  Enjoy 

•  Really  Did  Not  Enjoy 

10.  Overall,  was  your  visit  to  CSC  a  positive  one? 

•  Very  Positive 

•  Somewhat  Positive 

•  Neutral 

•  Slightly  Negative 

•  Very  Negative 

1 1 .  The  tiling  I  like  best  about  the  procedure  is: 


12.  One  thing  I  would  change  about  the  procedure  is: 


13.  Additional  comments  about  the  procedure: 
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7.3.8.3  Long  Form  Self  Report 

Project  #  303_1  (IRB  0004-1) 


Long  Form  Self  Report 

ID: _ _ 

PRE-EXPERIMENT 

Please  complete  this  form  either  answering  the  questions  or  placing  an  “X?’  on  the 
measuring  bars. 


My  PHYSICAL  fatigue  is: 


1  ,  1  1  1  1  1  1  1 

1  1 

1  1  1 

_1_L 

J_J_ 

L_U 

Low  10  20  30 

40 

50 

60 

70 

80 

90 

High 

My  MENTAL  fatigue  is: 

1  .  1  1  1  1  1  1.1 

1  1 

j_Lj 

lJU. 

_Li_ 

IjJ 

Low  10  20  30 

40 

50 

60 

70 

80 

90 

High 

My  OVERALL  fatigue  is 
1  1  1  1  1  1  1  1 1 

• 

• 

1  1 

1  1 

Li_L 

J-L 

1-U 

Low  10  20  30 

40 

50 

60 

70 

80 

90 

High 

Please  rate  your  overall  stress  level: 

1  1  1  1  1  1  1  1  1  1  1  1  1 

1_J_1 

1  1 

1_U 

Low  10  20  30 

40 

50 

60 

70 

80 

90 

High 

The  most  tired/strained  part  of  my  body  right  now  is  (please  circle  one): 


eyes 

head 

left  arm 

right  arm 

legs 

left  hand 

right  hand 

chest 

stomach 

not  really 

tired/strained 

How  comfortable  are  you  in  the  foUowing  regions  right  now? 

Head 

I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  '  I  I  I  I  I 

Low  10  20  30  40  50  60  70  80  90  High 

Arms 

I  .  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I 

Low  10  20  30  40  50  60  70  80  90  High 
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Chest/Torso 

I  I  I  I  I  1  I  '  I  '  I  I  I  I  I  I  1  I  I  I  1 

Low  10  20  30  40  50  60  70  80  90  High 


Legs 

u 

Low  10  20  30 


I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  i-i 


40  50  60  70  80  90  High 

Project  #  303_1  (IRB  0004-1) 


Long  Form  Self  Report 

ID:^ _ _ _ 

POST-EXPERIMENT 

Please  complete  this  form  either  answering  the  questions  or  placing  an  “X”  on  the 
measuring  bars. 


My  PHYSICAL  fatigue  is: 

1  1  1  1  1  1  1  1  L_ 

l1 

t  1  1 

_Ll. 

1j_ 

U- 

J 

Low  10  20  30 

40 

50 

60 

70 

80 

90 

High 

My  MENTAL  fatigue  is: 

1  ,  1  ,  1  1  1  1  1 

i_L 

1  1  1 

1  1  ,1- 

1_L. 

J 

Low  10  20  30 

40 

50 

60 

70 

80 

90 

High 

My  OVERALL  fatigue  is: 

1  ,  1  1  i  1  1  1  1 

_lJ_ 

L±±. 

_Lj. 

_Ll 

J 

Low  10  20  30 

40 

50 

60 

70 

80 

90 

High 

Please  rate  your  overall  stress  level: 

1  .  1  1  1  1  1  1  1  1  1  1  1- 

iJ_L 

i_L 

J 

Low  10  20  30 

40 

50 

60 

70 

80 

90 

High 

The  most  tired/strained  part  of  my  body  right  now  is  (please  circle  one): 


eyes 

head 

left  arm 

right  arm 

legs 

left  hand 

right  hand 

chest 

stomach 

not  really 

tired/strained 

How  comfortable  are  you  in  the  following  regions  right  now? 
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Head 


1  1 

1  1  1 

I 

1  1 

l_Ll 

±J- 

±J_ 

i_lJ 

Low 

10 

20 

30 

40 

50 

60 

70 

80  90 

High 

Arms 

1  1 

1  1  1 

1  1  1 

1  1 

iJ-L 

-i_L 

J_J_ 

1  1  1 

Low 

10 

20 

30 

40 

50 

60 

70 

80  90 

High 

Chest/Torso 

1  .  1  1 

1  1  1 

1  _L 

jJ. 

XJ-L 

J_L 

J_L 

IjJ  • 

Low 

10 

20 

30 

40 

50 

60 

70 

80  90 

High 

Legs 

1  1 

1  1 

1  1  1 

1  1 

1  1  -t 

J_i 

_Ll 

_LlJ 

1  I. 

Low 

10 

20 

30 

40 

50 

60 

70 

80  90  High 

Project  #  303_1  (IRB  0004-1) 

Long  Form  Self  Report 

D): _ 

DEBRIEFING 

Please  complete  this  form  either  answering  the  questions  or  placing  an  “X”  on  the 
measuring  bars. 


My  PHYSICAL  fatigue  is: 

I  I  I  I  I  I  I  '  I  I  I  I  I  I  I  I  I  I  I  i-i 

Low  10  20  30  40  50  60  70  80  90  High 

My  MENTAL  fatigue  is: 

I  I  I  I  I  I  I  i  I  I  I  I  I  I  I  I  I  I  I  i-i 

Low  10  20  30  40  50  60  70  80  90  High 

My  OVERALL  fatigue  is: 

I  I  I  I  I  I  I  '  I  I  I  I  I  I  I  I  I  I  I  -i-i 

Low  10  20  30  40  50  60  70  80  90  High 

Please  rate  your  overall  stress  level: 

I  .  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I 

Low  10  20  30  40  50  60  70  80  90  High 


183 


CYBERNET  SYSTEMS  CORPORATION 


now  is  (please  circle  one): 

right  arm  legs 

stomach  not  really 


How  comfortable  are  you  in  the  following  regions  right  now? 
Head 


1 _ L 

1  1 

1  1 

1  1 

1  1 

1  1 

L±J. 

J_L 

J_L 

J 

Low 

10 

20 

30 

40 

50 

60 

70 

80 

90 

High 

Arms 

1  1 

1  1  1 

1  1  1 

1  1 

,  1 

1  1  1 

lJ_l 

AJu 

i_L 

J 

Low 

10 

20 

30 

40 

50 

60 

70 

80 

90 

High 

Chest/Torso 

1  1  1  1 

1  1  1 

1  1 

,  1 

1  1 

1  1  1 

J_L 

1_L 

J 

Low 

10 

20 

30 

40 

50 

60 

70 

80 

90 

High 

Legs 

1  1 

1  1 

1  1  1 

1  1  1 

1  1 

1  1 

1  1  1 

Aj- 

J 

Low 

10 

20 

30 

40 

50 

60 

70 

80 

90 

High 

The  most  tired/strained  part  of  my  body  right 
eyes  head  left  arm 

left  hand  right  hand  chest 

tired/strained 


7.3.8.4  Short  Form  Self  Report 

(Note  that  for  every  experimental  configuration,  the  subject  was  afterwards  asked  to  fill 
out  this  form.  The  section  listed  under  ID,  currently  ‘STERNBERG  1  ’  was  changed  to 
indicate  the  appropriate  experimental  trial.) 

Project  #  303_2  (IRB  0004-1) 
Short  Form  Self  Report 

ID: _ 

STERNBERG  1 

Please  rate  the  computer  task  by  placing  an  “X”  on  the  line  that  reflects  the  level  of 
demand  for  each  workload  sub-category. 
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Mental  Demand 

I . I  I  I  I  I  I  I'l  I  I  I 

Low  10  20  30  40  50  60  70  80  90 

Physical  Demand 

. I  I  I  I  I  1  I  I  I  I  I  I 

Low  10  20  30  40  50  60  70  80  90 

Temporal  Demand 

I  ■  I  '  I  '  I  '  I  i  I  '  I  I  I  I  I  I  I  I'  I 

Low  10  20  30  40  50  60  70  80  90 

Performance 


Low  10  20  30  40  50  60  70  80  90 

Effort 

I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I 

Low  10  20  30  40  50  60  70  80  90 

Frustration 

I  .  I  .  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I 

Low  10  20  30  40  50  60  70  80  90 

Stress 

I  ■  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I 

Low  10  20  30  40  50  60  70  80  90 

How  bored  were  you  with  the  previous  task? 

I . .  I  I  I  I  I  I  I  I  I  I  I  I  iJ 

Not  Bored  10  20  30  40  50  60  70  80  90 


High 


High 


High 


High 


High 


High 


High 


Veiy  Bored 
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7.3.8.5  Task  Form  Self  Report 


Project  #  303_1  (IRB  0004-1) 


Task  Form  Self  Report 

ID; _ _ 

STERNBERG 

Please  circle  which  scale  was  a  more  important  contributor  to  workload  for  you  in 


the  previous  computer  tasks: 

1.  Mental  Demands 

or 

Frustration 

2.  Mental  Demands 

or 

Performance 

3.  Mental  Demands 

or 

Effort 

4.  Temporal  Demands 

or 

Frustration 

5.  Physical  Demands 

or 

Performance 

6.  Performance 

or 

Effort 

7.  Temporal  Demands 

or 

Mental  Demands 

8.  Physical  Demands 

or 

Temporal  Demands 

9.  Mental  Demands 

or 

Physical  Demands 

10.  Performance 

or 

Frustration 

11.  Physical  Demands 

or 

Effort 

12.  Effort 

or 

Frustration 

13.  Temporal  Demands 

or 

Performance 

14.  Physical  Demands 

or 

Frustration 

15.  Temporal  Demands 

or 

Effort 

A  <  «  A 

How  bored  were  you  with  the  previous  set  of  tasks? 


I  .  I . I  I  I  I  I  I  I  I  I  I  I  I  I 

Not  Bored  10  20  30  40  50  60  70  80  90  Veiy  Bored 
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Project  #  303_1  (IRB  0004-1) 
Task  Form  Self  Report 

ID:^ _ _ 

RECALL 


Please  circle  which  scale  was  a  more  important  contributor  to  workload  for  you  m 
the  previous  computer  tasks: 

1.  Mental  Demands 

or  Frustration 

2.  Mental  Demands 

or 

Performance 

3.  Mental  Demands 

or 

Effort 

4.  Temporal  Demands 

or  Frustration 

5.  Physical  Demands 

or  Performance 

6.  Performance 

or 

Effort 

7.  Temporal  Demands 

or  Mental  Demands 

8.  Physical  Demands 

or  Temporal  Demands 

9.  Mental  Demands 

or 

Physical  Demands 

10.  Performance 

or 

Frustration 

11.  Physical  Demands 

or  Effort 

12.  Effort 

or 

Frustration 

13.  Temporal  Demands 

or  Performance 

14.  Physical  Demands 

or  Frustration 

15.  Temporal  Demands 

or  Effort 

How  bored  were  you  with 

the  previous  set  of  tasks? 

I  I  h  I  '  I  '  I  I  I  I  I  I  I  I  I  I  I  J-l 

Not  Bored  10  20  30  40  50  60  70  80  90  Very  Bored 
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Project  #  303_1  (IRB  0004-1) 
Task  Form  Self  Report 

ID: _ _ 

TRACKING 


Please  circle  which  scale  was  a  more  important  contributor  to  workload  for  you  in 
the  previous  computer  tasks: 

1.  Mental  Demands 

or  Frustration 

2.  Mental  Demands 

or 

Performance 

3.  Mental  Demands 

or 

Effort 

4.  Temporal  Demands 

or  Frustration 

5.  Physical  Demands 

or  Performance 

6.  Performance 

or 

Effort 

7.  Temporal  Demands 

or  Mental  Demands 

8.  Physical  Demands 

or  Temporal  Demands 

9.  Mental  Demands 

or 

Physical  Demands 

10.  Performance 

or 

Frustration 

11.  Physical  Demands 

or  Effort 

12.  Effort 

or 

Frustration 

13.  Temporal  Demands 

or  Performance 

14.  Physical  Demands 

or  Frustration 

15.  Temporal  Demands 

or  Effort 

How  bored  were  you  with  the  previous  set  of  tasks? 

1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1 

Not  Bored  10  20 

30 

40  50  60  70  80  90  Very  Bored 
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8.  Future  Opportunities 

This  Phase  11  effort  produced  a  ^stem  and  validated  methodology  for  die  comprehensive 
assessment  of  human  performance  in  a  wide  range  of  human  operator  task  environments. 
Now  that  this  unique  tool  is  available,  there  are  extensive  opportunities  for  further 
research,  scientific  study,  and  development.  The  developed  system  can  facilitate  research 
protocols  and  studies  that  were  previously  difficult,  awkward,  or  ineffective.  This  is 
particularly  true  with  regard  to  die  combined,  synchronized  measurement  of  different 
types  of  performance  related  parameters.  For  instance,  the  ability  to  acquire  brain  wave 
activity  and  eye  tracking  data  within  an  integrated,  synchronized  data  collection  system 
will  be  of  tremendous  value  for  a  wide  array  of  cognitive  research  studies. 

Continued  research  and  development  efforts  could  help  support  the  establishment  and 
setup  of  new  experimental  protocols  that  take  advantage  of  the  developed  system.  Given, 
the  focused  goal  of  such  an  experiment,  the  system  could  be  enhanced  and  tailored  to 
meet  very  specific  needs  of  the  targeted  protocol.  For  example,  additional  signal  viewers, 
data  analysis  functions,  and  other  custom  “add-on”  features  can  be  readily  developed  to 
tailor  the  system  performance  to  a  given  study  or  application.  Cybernet  has  the  expertise 
to  support  the  establishment  and  execution  of  research  protocols  using  the  developed 
human  performance  measurement  system. 

In  addition,  there  is  significant  opportunity  to  produce  commercial  products  from  the 
developed  technology.  Each  individual  component  of  the  human  performance  based 
measurement  system  has  product  development  plans  already  underway.  For  instance,  the 
eye  tracking  technology  is  being  integrated  into  a  head-mounted  display  (along  with  voice 
recognition)  for  hand-free  control  of  a  wearable  computer  interface.  The  eye  tracking 
technology  also  has  product  potential  for  medical  applications.  Furthermore,  the 
physiological  measurement  system  technology  and  related  software  is  part  of  Cybernet’ s 
business  plan  for  bringing  medical  data  monitoring  applications  to  the  Internet. 
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